PaperlayerView exact revision on arXiv

Source-linked AI summary

Interface-Induced Trajectory Censoring

Wenbo Wang

arXiv:2609.03966v1cs.AIcs.LG

TL;DR

The paper asks whether reported tool-call rates measure model capability when serving interfaces can censor emitted calls. It measures this across benchmarks, model scales, interfaces, and RL rollouts, finding that the observed rate belongs to the model–interface stack and that repair restores mechanism more reliably than outcomes.

  • Problem

    Tool-call rates read from serving stacks can conflate model incapability with interface-induced trajectory censoring.

  • Method

    The paper compares fixed models, tasks, decoding, and seeds across serving adapters, templates, parsers, scales, benchmarks, and RL rollout collection.

  • Results

    0.00-to-0.96 BFCL swing: template–parser interaction determined the reported score, while repaired interfaces restored parsed calls and tool-mediated trajectories.

  • Takeaways & Limitations

    Observed tool-call rates should report the model-interface composition, including emitted-but-unparsed calls and exact serving configuration.

  • Takeaways & Limitations

    The training evidence lacks a clean repaired-FC control, and the scale ladder is a within-family Qwen2.5-Coder observation rather than a general law.

Abstract

from arXiv · show

Agent evaluations report a tool-call rate read off the serving stack. That number can be zero while the model is emitting well-formed calls: the interface censors the trajectory before anything downstream sees it. On BFCL v4's own data, executor and scorer, holding weights, cases, decoding and seeds fixed and changing only the serving adapter, the same model scores 0.00 or 0.96 / 0.19. A 2x2 over chat template and parser locates the effect exactly: both main effects are exactly zero and all of it sits in the interaction -- no component is defective, and repairing one side of the contract buys precisely nothing. On tau-bench's 115 interactive retail tasks the same swap moves server-parsed calls from 0 to 636 and tasks reaching any tool execution from 0 to 103. Our probe reproduces the funnel across a 21x scale range of Qwen2.5-Coder: the server parses 0/100 at every size while well-formed emitted calls rise to 80/100 at 32B (~72 after calibration against an adjudicated gold standard). Under a matched envelope, across a comparable scale span, the silent fraction stays at 0-2, a prediction committed to the repository before the run. Llama-3.1-8B's 23% rate of calling the task function itself as a tool falls to 0 under one strict:true flag. The mismatch reaches inside the training loop, and its consequence is scale-dependent: in verl's AgentLoop at 7B, 45 of 115 generations carry a complete call; 0 are accepted, 0 execute, 0 return an observation. At 1.5B the same zero is over-determined, so we report the two scales separately. At evaluation time, repairing the adapter restores the mechanism but not a significant outcome gain: parsing 0->84, rescues 0->9, pass rate 53->62 (n.s.). We release a 98-line preflight check that catches every silent failure here. The observed tool-call rate is not a property of the model alone; it is a property of the model-interface stack that measures it.

1 Introduction

The paper identifies interface-induced trajectory censoring as the source of apparently absent tool use: models can emit valid calls that the serving stack never exposes downstream. It frames tool-call rates as properties of a model–interface contract rather than the model alone.

  • Motivation: 2.6–2.8 points: pass@1 rose across three Qwen2.5-Coder-1.5B runs, but none reached significance at α = .05.The authors investigated the training curve because newly passing items were overwhelmingly solved on the first turn.
  • Motivation: 91–94% of newly-passing items passed on the first turn, while turn-2-or-later rescues stayed at 6–9 of 540.The server returned HTTP 200 with an empty tool_calls array, and the training loop recorded a single-turn trajectory.
  • Core concept: Interface-induced trajectory censoring is systematic unobservability caused by a boundary in the model × protocol × serialization × parser × execution stack.A parser mismatch can make model incapability and interface failure produce the same observed output: an empty tool-call count.
  • Diagnosis: 4254 archived first turns produced zero cases where vLLM’s hermes extractor failed to parse a call accepted by its own rules.The paper therefore locates the failure in independently configured emission, templating, and parser contracts rather than in parser implementation.
  • Core concept: Qwen2.5-Coder emits bare calls without the hermes envelope, whereas Qwen2.5-Instruct emits the envelope but can break its JSON.The resulting observable at multiple boundaries is HTTP 200 with an empty tool_calls array.
  • Evidence: 0.00 to 0.96 / 0.19: changing only the serving adapter moved BFCL scores, while τ-bench parsed calls moved from 0 to 636.A 2 × 2 over chat template and parser found both main effects exactly zero, with the effect in their interaction.

2 Related Work

Related work examines evaluation validity, parsing-induced measurement error, evaluator alignment, and constrained tool-use protocols. This paper extends those concerns to serving-stack contracts, scale-dependent distortion, and reinforcement-learning trajectories.

  • Evaluation validity: Prompt formatting, single-prompt evaluation, and benchmark configuration can change reported results by margins comparable to model differences.These studies vary evaluator-controlled inputs, whereas this paper isolates a serving-stack parser as a measurement component.
  • Parsing-induced measurement error: 0% to 76%: a corrected fuzzy parser recovered security-threat accuracy from identical model outputs in the closest prior parsing study.The present work adopts the fixed-bytes, different-rules standard while extending analysis across layers, families, scale, and RL rollouts.
  • Audits and failure taxonomies: 18.5% evaluator–human misalignment was reported across BFCL v4, τ 2-bench, LiveMCPBench, and MCP-Atlas.Other concurrent work also documents ground-truth defects in tool-use benchmarks.
  • Constrained decoding and protocols: strict: true eliminates a 23% role-confusion failure in the paper’s Llama counter-instance, although added reasoning worsened the CoT control.The paper compares this with ReAct and guided-decoding machinery used by strict and required modes.
  • Prior report of the Qwen case: vLLM issue #32926 documented Qwen2.5-Coder’s missing tool-call tokens and reported 98–100% compliance with a dedicated parser and few-shot template.This paper reproduces the zero-tag baseline and adds scale quantification, additional families, and training consequences.

3 Setup

The experimental setup uses paired 100-item code tasks, ReAct and function-calling protocols, RL training, documented vLLM serving configurations, and an independent classifier for emitted calls. Validation and admissibility rules separate model behavior from serving and measurement failures.

  • Tasks: 100 problems: every full-length arm uses the identical decontaminated KodCode item set, enabling paired cross-arm comparisons.The unique item-set count is 1 across all 50 full-length arms.
  • Protocols: ReAct uses a fixed text protocol, while function calling uses OpenAI-style tool_choice: auto with three schema variants.The variants are terse, rich, and strict, with strict: true and additionalProperties: false.
  • Training and serving: GRPO and RLOO provide the training algorithms, with vLLM 0.27.1 and per-family parser and chat-template configurations used for serving.Both protocols use max_tokens = 2048; temperature=0 is greedy but not bit-wise deterministic under continuous batching.
  • Emission measurement: Three nested classifier tiers measure emitted calls independently: tight, strong, and weak.The tight tier requires a run_tests call whose arguments.code is a string literal containing real Python and excludes observed contaminants.
  • Validation: κ = 0.871 [0.756, 0.958]: adjudicated validation supported the classifier used to measure emitted calls.The gold standard covered all fifteen non-unanimous items selected by three-way disagreement.
  • Instrument control: tool_choice: auto versus required isolates serving acceptance: the former returned zero parsed calls while the latter returned one valid run_tests call.The control shows that the pipeline can accept a conforming call without establishing that the model emits one under auto.
  • Admissibility: Exactly 100 items, exit code 0, zero request errors, and matching provenance are required for pass-rate comparison.Arms with request errors remain valid for error-rate census but are marked N/A for pass rates.

4 Results

Across benchmarks, scale probes, training rollouts, and controlled interface swaps, tool-call censoring is produced by mismatched model-interface contracts rather than isolated parser defects. Repairing the serving path restores parsed and executable calls, while outcome gains remain limited or nonsignificant in the reported evaluation.

  • 4.1 BFCL v4: 0.00 versus 0.96 / 0.19 BFCL scores result from changing only the serving adapter, with the executor, scorer, cases, decoding, and seeds fixed.The 2 × 2 shows both main effects exactly zero and all effect in the template–parser interaction.
  • 4.2 τ-bench: 0 versus 636 parsed calls and 0 versus 103 tasks reaching tool execution occur across 115 τ-bench retail tasks after the adapter swap.The two arms differ in conversation length by construction, so quantities are reported per task rather than per request.
  • 4.3 Censoring against scale: 0/100 server-parsed calls persist across a 21× Qwen2.5-Coder scale range while well-formed emitted calls rise to 80/100, approximately 72 after correction.Under a matched envelope, the silent fraction stays at 0–2; the parsed column is not monotone, so the result establishes presence rather than a trend.
  • 4.5 Schema-layer failure: Strict:true changes valid execution from 73→97 and final task performance from 44→61 while eliminating a 23% role-confusion failure.With strict decoding, both protocols have identical execution mechanics, so the remaining turn-1 difference concerns first-draft code quality rather than tool-use mechanics.
  • 4.7 Evaluation repair: 0→84 parsing and 0→9 rescues restore the mechanism, but pass rate changes 53→62 without statistical significance.The two FC arms have identical turn-1 pass rates, 53 versus 53, providing the internal-validity check for downstream adapter effects.

5 Discussion

The paper argues that tool-use evaluation and training must report the model-interface composition, verify serving configurations before experiments, and repair interfaces before comparing protocols. Silent trajectory censoring can prevent tool branches from receiving learning signal when tool-free rewards remain available.

  • Tool-call evaluations should report parsed calls, emitted-but-unparsed calls, request errors, and the exact serving configuration.
  • A 98-line pre-flight script checks that canonical tool requests return non-empty, correctly named, parseable tool calls.
  • The guidance for parser configuration is absent from surveyed model cards despite their recommending vLLM, leaving the pairing undocumented at the point of use.
  • Serving configuration must be exhausted before protocol comparisons, because uncorrected interfaces can make those comparisons measure the stack instead.
  • When tool trajectories are censored at rate ∼1 and rewards permit tool-free paths, the tool-using branch has no sampled return to reinforce.

6 Limitations

The paper’s conclusions are constrained by limited controls, incomplete rollout records, multiple comparisons, and several dataset, task, scale, and validation boundaries.

  • Scope boundaries: The scale ladder supports only a within-family observation, not a general claim that censoring increases with model size.The ladder uses Qwen2.5-Coder, one mismatched interface, and one task; a second family is needed to test generality.
  • Training evidence: There is no clean repaired-FC training control, so the paper does not causally establish that censoring prevents RL from learning multi-turn repair.The training comparison changes protocol and interface together.
  • Instrumentation: Observed zero executions do not distinguish zero attempts from attempted calls that were discarded, because raw rollout text was not saved.The instrumentation records sampled experience, not the policy’s full support.
  • Statistical interpretation: Conditioning on successful parses is post-treatment conditioning, and the residual Qwen difference is not statistically conclusive.The reported residual is +8.4 pp with 95% CI [−0.5, +17.4] and p=0.118.
  • Statistical interpretation: After roughly a dozen tests, only the strict:true intervention and 300-item protocol gap survive the Bonferroni threshold.The threshold is approximately 0.0042; the paper makes no claim on the listed nonsurviving results.
  • Scope boundaries: The study is additionally limited by one task family and tool, non-random samples, single-sample headline arms, scale stopping at 32B, and validation constraints.The appendix also notes asymmetric taxonomy, ReAct-only sufficiency testing, and a 4000-character output cap.

7 Conclusion

The paper concludes that serving interfaces can silently censor tool-use trajectories and that the measured effect belongs to the model–interface contract rather than the model or parser alone. Interface repair restores tool-call processing but only partly closes the outcome gap, while reproducibility depends on corrected provenance and admissible comparisons.

  • Core conclusion: 45 of 115 training generations carried complete calls, yet none became an action, executed, or returned an observation.The paper interprets this as no tool-mediated trajectories in the observed rollout distribution.
  • Core conclusion: A 2 × 2 control found both chat-template and parser main effects exactly zero, with the 0.00-to-0.96 swing entirely in their interaction.A matched-interface scale ladder held the silent fraction at 0–2 while the mismatched interface reached 80.
  • Measurement consequence: Human validation missed 12 of 98 tool calls after fenced code blocks, reinforcing that server-parsed rates are unsafe as direct measures of model capability.The same bytes led annotators and parsers to conclude that no call had been made.
  • Data integrity: The reported provenance included incorrect max_tokens values, but corrected comparisons remained conservative because ReAct had the smaller generation budget.Re-running three main ReAct arms at true 2048 changed final pass by at most one item.
  • Data integrity: The corrected run_tests initiation rate for Llama-3.1-8B was 74/100 after excluding 23 calls aimed at other task functions.Those 23 calls were reproduced as wrong-tool calls in a matched re-run.
  • Data integrity: Pass-rate comparisons are inadmissible for arms with missing data, and an earlier p = 0.648 result is retracted.The error-rate census remains valid, but no p-value is computed for those arms.

B Reproducibility

The release provides scripts for analysis, figures, probing, and per-arm validation, with pinned model snapshots and serving configurations recorded for reproducibility.

  • Analysis: The analysis scripts generate intent counts, final tables, variance and error-rate tables, and figures without GPU access.The listed commands are analysis/intent.py, final_table.py, and make_figs.py.
  • Probing: The probe records trajectories across React or FC protocols, optional or mandatory strength, FC schemas, parser adapters, temperature, seed, and output path.The probe requires a vLLM server and accepts 100-item runs through command-line flags.
  • Validation: Per-arm validation checks line counts, return codes, errors, provenance, and script hashes.The validation command is validate_arms.py.
  • Provenance: Model paths, HF revisions, and serving flags are recorded because tokenizer configuration inheritance can change family behavior across repository revisions.Snapshots must therefore be pinned for reproducible capability checks.
  • Release contents: The repository contains all 50 full-length arms, training logs, errata, and the per-configuration index.It also records the third-party parser version and rewritten few-shot examples with both hashes.

C Prompts and tool schemas, verbatim

The experiments compare ReAct and function-calling prompts, optional and mandatory tool use, and terse, rich, and strict schemas. They also define the emitted-call criterion and document serving and training instrumentation.

  • C.1 ReAct: Optional prompts permit direct answers, whereas mandatory prompts require an initial tool action; the main comparison uses matched optional strength.
  • C.1 ReAct: ReAct presents Thought, Action, Action Input, Observation, and Final Answer, while function calling exposes run_tests as the available tool.
  • C.3 Tool schemas: terse, rich, strict: The terse schema specifies run_tests with one string parameter, code, while the rich schema additionally warns that the tool is not the task function and supplies an example.
  • C.3 Tool schemas: terse, rich, strict: 22/100 items still pass task-function parameters under the explicit rich warning, with the same parameter names that failed under terse.
  • C.3 Tool schemas: terse, rich, strict: strict adds "strict": true and takes the wrong-tool rate to zero, whereas adding a Thought requirement raises it from 23 to 59.
  • C.5 The role-disambiguation few-shot (§4.8): The role-disambiguation few-shot increases recoverable calls from 52 to 64, leaves run_tests calls at zero, and lowers final pass from 15 to 13.
  • C.6 The intent criterion: The tight emitted-call criterion requires a run_tests call whose code argument is a string containing real Python; strong and weak tiers extend this classification.
  • D.1 Per-family serving flags: The preflight check asserts a nonempty, correctly named, parseable tool call and repeats with tool_choice: required; it would have caught every silent failure, while verl's named tool_calls metric is elapsed seconds.

D.5 The four obstacles to a repaired-FC training arm

Attempts to create a repaired function-calling training arm encountered parser-isolation, semantic role-confusion, and unavailable constrained-decoding obstacles. The corrected account treats the repair as a verl parser-registration change rather than a vLLM plugin or forced format.

  • Four successive attempts to construct a repaired-FC RL condition were reported, with three initial explanations corrected after inspection.
  • A vLLM parser plugin does not reach verl's AgentLoop because the training path selects verl's independent hermes parser.
  • A verl-side parser raises recoverable calls from 0/100 to 52/100 at 1.5B, but zero name run_tests and all target the task function.
  • Role-disambiguation increases recoverable calls from 52 to 64 while run_tests calls remain zero and task pass falls from 15 to 13.
  • Constrained decoding is unavailable in verl 0.9.0's vLLM rollout path, and forcing tool choice would impose a format rather than accept the model's existing output.
  • E Arm inventory: All full-length arms share one item set, while seven have known-wrong max_tokens records and five are inadmissible for pass rates.

F Human validation of the intent criterion, in full

The intent criterion was validated through stratified human review of 98 outputs, with deliberate attention to the decision boundary and to failures in the first annotation procedure.

  • 98 outputs were sampled by classifier verdict—40 tight, 28 strong-but-not-tight, and 30 neither—to estimate false positives and false negatives near the boundary.
  • The first round's 12 false positives comprised 11 reading-order failures and one truncation caused by a 2400-character annotation-pack limit.
  • The first adjudication was one-sided: selecting all 13 items by disagreement with the classifier made the increase from κ = 0.713 to 0.936 non-diagnostic of reliability.

A second independent annotator

A second blind annotation round corrected the first procedure's truncation and selection problems, producing high agreement on identical bytes and a calibrated classifier estimate.

  • The second annotator reviewed the same 98 items blindly, while full outputs and a pre-specified rule covering all three-way disagreements replaced the earlier procedure.
  • 11 of 12 raw annotator disagreements involved outputs longer than 2400 characters, and the annotators agreed on 61 of 62 items with identical bytes, yielding κ = 0.967.
  • All 15 three-way disagreements went to a third adjudicator under the preregistered rule, which moved away from the classifier on 6 of 15 cases.
  • The adjudicated tight-stratum precision is 36/40 = 0.900, correcting the 32B headline from 80/100 to approximately 72 without changing the scale trend.
  • The validation estimate changed from 0.950 to 0.975 and then 0.900 as one-sided selection and single-rater reliance were replaced by adjudicated gold labels.
  • Because outputs are capped at 4000 characters, emitted-call counts are lower bounds when calls extend beyond the stored bytes.
  • Figure 2 shows pass@1 rising through first drafts while rescues after turn 2 remain at 6–9 of 540, and no tool call succeeds in those runs.

G.2 Result tables referenced from §4

The result tables trace tool-call failures to interface contracts and quantify their effects across benchmarks, model scales, training, and evaluation. Offline replay and controlled comparisons show that calls exist but are lost before or at format acceptance, while adapter repair restores parsing without uniformly improving outcomes.

  • Failure-layer decomposition: 0 genuine parser losses appear across all arms and 4254 de-duplicated first turns; Coder fails at the envelope, whereas Instruct fails inside the payload.The replay separates absent envelopes, malformed JSON, trailing text, and genuine parser loss.
  • BFCL adapter comparison: 0/200 calls are parsed in the hermes arm, while the repaired arm parses 196–197 first turns and executes 98–99, with 19 successes in both runs.The hermes arm is identical across runs; intermediate repaired-arm counts vary by one case.
  • Scale probe: 0 server-parsed calls occur at every scale, while well-formed emitted calls reach 80/100 at 32B and 72 after applying the 0.900 correction factor.The bare-JSON and tight offline columns show that calls exist despite the zero hermes column.
  • Scale-dependent adapter effects: 84/100 calls are recovered at 7B but only 1/100 at 3B; the 3B checkpoint emits zero <tools> tags in 100 items while finishing at pass rate 53.The dedicated solution changes parser, chat template, and few-shot examples simultaneously.
  • Controls: 0/100 parser-accepted calls remain after making the tool instruction mandatory, while final pass falls from 31 to 15 and unparsable outputs rise to 46/100.Adding a reasoning scaffold likewise raises role confusion from 23 to 59; neither intervention changes the interface.
  • Sampling variance: 72/72/72 final passes across ReAct seeds conceal pairwise Jaccard overlap of only 0.71–0.80, while 85 of 542 first-turn completions change between training checkpoints.Item-level set comparisons reveal policy movement that aggregate totals conceal.

H Limitations, in full

The evidence is bounded by task, model-family, serving-stack, repository, and experimental-control limitations. These constraints limit generalization and causal interpretation without negating the reported interface findings.

  • Scope of evidence: The scale and training results use one task family, while the scale ladder is Qwen2.5-Coder alone under one mismatched interface.The paper explicitly does not claim censoring is generally scale-increasing or equate parameter count with capability.
  • Causal interpretation: The training comparison lacks a clean repaired-function-calling control because it changes protocol and interface together.ReAct is treated as a positive-control interaction channel rather than a parser-repair control.
  • Measurement boundaries: Emitted-call counts are lower bounds because stored raw output is capped at 4000 characters, and the sample estimates per-stratum rather than population rates.These limits affect classifier and annotator visibility and the interpretation of estimated rates.
  • Environment dependence: All measurements use vLLM 0.27.1 and verl 0.9.0 without replication on a second serving stack, so version-specific behavior remains unresolved.Parser behavior, template defaults, and strict:true semantics are properties of the pinned versions.
  • Needed validation: The proposed follow-up requires a repaired-FC training comparison at 7B and additional standard benchmarks, with rescue arms at small sizes.A checkpoint that never attempts a call would produce an uninformative zero rather than a censored result.
Loading 2609.03966v1…