Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation

1 Introduction

• Phi-3 and Gemma-3 are reported at 0% under native function calling even though every turn is a rejected request and the models never ran.The harness records rejection and non-response as ordinary non-call turns, which can make serving failures look like model behavior.Source: §1 · p. 2
• For accepted models, adding a text tool list while retaining the native channel recovers most fidelity, whereas a uniform text protocol lowers fidelity for Llama-3.2.The best configuration is model-dependent.Source: §1 · p. 2
• Ollama, llama.cpp, vLLM, and SGLang handle the identical tool-call request differently, making the outcome a serving-stack property rather than a model property.The cross-stack check finds the same weights rejected by Ollama run on llama.cpp.Source: §1 · p. 2
• Turn-pooled and per-seed fidelity estimates differ by up to about 55 points when one looping episode dominates the pooled rate.The divergence occurs on thin denominators.Source: §1 · p. 2
• Constrained decoding makes every model emit a valid call by construction, but weaker models can loop without terminating, shifting rather than removing the failure.Source: §1 · p. 2
• At this measurement precision, the experiments do not support claims about scale, model family, or reasoning.Source: §1 · p. 2

Function-calling evaluation

• BFCL distinguishes native function calling from prompting mode, but evaluates through per-model handlers and reports aggregate accuracy rather than per-turn outcomes on a held-fixed stack.The paper’s text-tools condition is prompting mode, while its control arm lies between the two.Source: §2 · p. 2
• AgentBench, τ-bench, and ToolSandbox study richer multi-turn or stateful interactions, but target hosted or fixed endpoints rather than local serving-stack effects.τ-bench also finds tool-use behavior unreliable across repeated trials.Source: §2 · p. 2

Serving backends and harnesses as measurement variables

• Pape et al. (2026) and Harness-Bench (Yao et al., 2026) treat backend or harness choice as a source of outcome variance, but not the local per-model tools= gate or its recording as a model non-call.The paper identifies controlled isolation of those mechanisms as its specific contribution.Source: §2 · pp. 2–3

Constrained decoding as the local remedy

• Constrained decoding is an established remedy for unparseable calls, but this paper uses it as a baseline rather than proposing a fix.Prior work includes grammar- or schema-guided decoding and finite-state decoding for tool calls [Zhang et al., 2023].Source: §2 · p. 3

Tool hallucination and failure taxonomies

• PA-Tool names hallucinated absent tool names as “schema misalignment,” while MAST catalogs agent failures without addressing the local serving layer.The paper questions the implicit assumption that the serving layer is transparent.Source: §2 · p. 3

Background: the tools= request and the serving layer

• A tools= request supplies function specifications, but the serving layer renders them into the model prompt and parses generated text into a structured tool_call.Template and launch configuration, rather than the model weights, determine whether the server performs these steps.Source: §3 · p. 3

Harness and per-turn taxonomy

• The harness labels turns as valid in-schema calls, hallucinated calls, unparseable text, no-call prose, or non-responses, and defines protocol fidelity over turns the model actually produced.It reports turn-pooled and per-seed rates with seed-level bootstrap 95% confidence intervals over eight task-instance seeds.Source: §3 · p. 3

Three serving conditions

• The three conditions are native tools=, native+hint with text guidance added, and uniform text-tools with tools= dropped.Native+hint is only defined for server-accepted models; its fixed prompt adds tool descriptions, JSON call format, and allowed names.Source: §3 · p. 3

Models

• The local model set spans default laptop-scale Ollama checkpoints and mixes models with and without native tool training to expose per-model gating.The study also includes a considerably newer cloud model as a high-capability anchor.Source: §3 · pp. 3–4

Native FC mismeasures, two ways

• Under native FC, Phi-3 and Gemma-3’s apparent 0% fidelity represents rejected requests on every seed, so the model never ran and the rate is undefined.Gemma-3-4B also produces one valid call per seed before non-response, which a naive denominator would count against it.Source: §4 · p. 4

Prompt guidance and serving conditions

• Qwen 0.5B–14B per-seed fidelity rises from 0/38/23/59/60% native to 35/58/82/89/80% native+hint, while Llama-3.2 falls from 82% native+hint to 44% text-tools.For accepted models, the text hint recovers much of the fidelity; the best-performing interface is model-dependent.Source: §4 · p. 4

Cross-stack handling

• The same tools= request is rejected by Ollama for some weights yet runs on llama.cpp; vLLM refuses it by default, while SGLang accepts it but returns calls as text.Even with parser flags enabled, the probed vLLM models did not surface native tool_calls.Source: §4 · p. 4, Figure 2 · p. 4

Rate aggregation

• Qwen-0.5B text-tools fidelity is 34% per-seed versus 85% turn-pooled, as one 41-turn episode dominates seven episodes that fail within one or two turns.Small denominators also yield wide intervals, including 34% [9, 59] for Qwen-0.5B text-tools.Source: §4 · p. 4, Table 1 · p. 5

Constrained decoding

• Constrained decoding yields valid in-schema calls on 8 of 8 single-step trials for all nine local models, but weaker models enter 600–900-turn loops when forced to call every turn.Qwen-0.5B reached 869 assistant turns in one seed before cutoff.Source: §4 · p. 5

Replication on a second task

• On the dependency-chain task, native gating replicates, but Llama-3.2 text-tools fidelity is 12% per-seed versus 60% on aggregation and the estimates remain noisy.Gemma-3-4B again emits a valid call and then fails to respond.Source: §4 · p. 5

Replication on HumanEval

• On HumanEval, native gating again rejects Phi-3 and Gemma-3 while accepting Qwen and Llama-3.2.The passage frames each benchmark problem as a tool-call task in which the model submits its solution.Source: §4 · p. 5

What we do not claim

• Overlapping intervals and pooling-dependent estimates do not support claims about scale, family, tool-training effects, or dissociation from reasoning.The small no-tools reasoning probe was inconclusive and omitted.Source: §4 · p. 5

• Ollama’s HTTP 400 reaches the wrapper, but rejection or retry exhaustion becomes a generic assistant error string without a persisted error type.The transport failure is detectable, but the saved trajectory retains only the message stream.Source: §5 · pp. 5–6
• A downstream analysis can attribute that ordinary non-call turn to the model unless it string-matches the harness’s specific error text.The paper separated rejection and non-response from model behavior using such a match.Source: §5 · p. 6
• Preserving structured error metadata is a checklist item, but better logging does not remove the serving stack’s per-model tool gating.The figure likewise shows that identical requests can produce different outcomes across stacks.Source: §5 · p. 6, Figure 3 · p. 6

6 Reproducibility

• The primary Ollama and llama.cpp runs were local and CPU-only, while vLLM and SGLang probes used GPU serving setups.The vLLM probe used a T4; the SGLang probe used an A800 80GB.Source: §6 · p. 6
• Ollama’s per-model tools= gate and HTTP 400 contract are release-specific; the native-mode results are scoped to version 0.30.8.Which model tags are gated may change in later releases.Source: §6 · p. 6
• Gemma-3-4B’s apparent all-valid outcome under uniform text-tools includes half non-responses, which fidelity over produced turns excludes.The figure therefore pairs fidelity with the non-response share.Source: Figure 4 · p. 7
• With the same Ollama release, rejection, cross-stack contrasts, and prompt-recovered fidelity reproduce, but sampled per-seed magnitudes do not reproduce to the digit.Temperature was 1.0, and seeds indexed task instances rather than the sampling RNG.Source: §6 · pp. 6–7

7 Discussion: why this will get worse

• Harnesses, protocol adapters, serving engines, and chat templates can each accept, rewrite, or reject requests independently of model ability.The text suggests that standards such as Model Context Protocol may add another translation step, but labels this conjecture unmeasured.Source: §7 · p. 7
• For standardized model comparisons, hold the serving interface fixed; for deployable-system evaluations, report the full model–server configuration.These evaluation targets should not be conflated.Source: §7 · pp. 7–8
• Across four stacks, the same tools= request receives different handling, including rejection, text calls, and native calls.Table 2’s probes cover identical GGUF weights on Ollama and llama.cpp, but only Qwen-0.5B and Phi-3 on vLLM and SGLang.Source: Table 2 · p. 8
• Report stack, version, and tool-call parser, and log rejected requests, timeouts, or empty responses separately from model non-calls.The recommended protocol also calls for per-seed rates with intervals rather than a single turn-pooled number.Source: §7 · p. 8
• When diagnosing a 0% tool-call rate, check for serving refusal or empty responses first, then parser configuration, before attributing failure to the model.An accepted request can still surface its call as text if the parser is not configured.Source: §7 · p. 8
• A second dependency-chain task reproduces native gating and the pooled-versus-per-seed gap, but its small-sample magnitudes are noisy and task-dependent.Table 3 reports four seeds; Gemma-3-4B’s 100% is based on only four produced turns amid non-responses.Source: Table 3 · p. 8

8 Conclusion

• A local server can gate requests and record refusals or empty responses as ordinary non-calls, yielding 0% for a capable model that never ran.The conclusion recommends holding the serving interface fixed for standardized comparisons, separating serving failures, and reporting per-seed rates with intervals.Source: §8 · p. 8

Limitations

• The four-stack check establishes differing request handling, but per-seed fidelity transfer across stacks is untested.vLLM and SGLang probes cover only Qwen-0.5B and Phi-3, so those default-handling results are specific to the tested model–stack pairs.Source: Limitations · p. 9
• Per-seed magnitudes are task-dependent: fidelity was measured on two Ollama tasks, with a single-turn HumanEval replication.The authors report that gating and number fragility replicate across the three tasks, but do not generalize the magnitudes.Source: Limitations · p. 9
• Thin denominators make pooled and per-seed rates diverge; weaker models produced only 8–16 turns across eight seeds.The authors report both estimates.Source: Limitations · p. 9
• The constrained-decoding baseline is single-turn, because its agentic version did not terminate on weak models.It demonstrates parseability recovery, not multi-turn behavior under constraint.Source: Limitations · p. 9
• Prompt-driven recovery is specific to the tested text hint, which includes an allowed-name list; a weaker hint might recover less.Source: Limitations · p. 9
• The authors omit an inconclusive reasoning probe and make no reasoning claim.Source: Limitations · p. 9
• An earlier three-seed pilot’s clean curve dissolved with eight seeds, indicating high variance in small-model rates.Source: Limitations · p. 9