Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models

1 Introduction

• The 600M reproduced valid, generalized tool calls on 6/6 examples, while the 1B scored 0/4–6 at every tested checkpoint despite nearly identical lenient harness scores.The harness credited tool mentions in prose rather than actual calls.Source: §1 · p. 1
• The 1B assigned only 10^-4–10^-5 probability to <|tool_call|> at the call’s first-token position, indicating a missing format prior despite its ≈6B-token tool-SFT phase.The 600M emitted the token greedily with only an ≈0.1% in-mix instruction signal.Source: §1 · p. 2
• A diverse-corpus recipe with 5× higher learning rate repaired the 1B in 2,202 steps and ≈3.3 hours, passing novel prompts with correct tool selection and unseen arguments.It reproduced 4/4 examples; on eight novel prompts it selected the right tool on all seven that required one and suppressed the call on the conversational prompt.Source: §1 · p. 2
• The repair left the trigger-token embedding essentially unmoved while late-layer attention weights changed, supporting a routing—not a “dead embedding”—account of the repair.97.7% of the embedding table was bit-identical; the recipe’s nominally unfrozen embeddings were numerically inert.Source: §1 · p. 2

1.1 Contributions

• The matched-pair thesis is that bootstrap composition influences whether structured output emerges by default or must be engineered later, rather than making the capability impossible to add.The authors identify the 600M as code-heavy and the 1B as web-heavy; they also caution that the natural experiment has confounds and the repair is one recipe.Source: §1 · p. 2, §1.1 · p. 3
• The paper proposes a minutes-of-CPU diagnostic ladder—reproduction, first-token probes, novel prompts, and embedding drift—to detect false positives, localize failures, and verify repairs.The authors propose it as a standing gate for small-model tool-use claims.Source: §1.1 · p. 2

Tool use in language models

• Toolformer, Gorilla [14], and ToolLLM study learning and evaluating API invocation; this paper instead asks whether sub-1B models produce the structured-call format at all.The authors note that common SFT recipes assume a pretrained base already has a structured-output prior.Source: §2 · p. 3

Code in the pretraining mix

• Prior work links code-rich pretraining to broader capabilities, while this paper presents a small-scale security-domain comparison in which the code-heavy model succeeds at structured generation and the web-heavy model does not by default.The proposed mechanism is a token-distributional prior over bracketed key–value continuations, not broader reasoning.Source: §2 · p. 3

Evaluation validity for generative benchmarks

• The Nano paper [17] documented keyword coincidences being over-credited on a classification metric; this paper reports the same failure class for tool use when free prose mentioning a tool is scored as a call.Its strict check requires a generalized, non-identical argument rather than exact reproduction of training data.Source: §2 · p. 3

The VectraYX series

• The VectraYX-Nano [17] and Vision-1B [18] papers provide the series context; this paper documents the 600M model and sharpens the picture of the 1B backbone’s tool-calling deficit.The paper says the deficit predated the vision phases and was repairable after localization.Source: §2 · p. 3

3 Architecture: A Matched Pair

• The models share decoder implementation, block structure, positional scheme, sequence length, tokenizer, and reserved token layout; only width and depth hyperparameters differ architecturally.Table 1 presents the two configurations side by side.Source: §3 · p. 3, Table 1 · p. 4

3.1 Shared tokenizer and special-token layout

• A tool call is defined by emitting ID 8, followed by a JSON object and ID 9, so the shared token layout rules out tokenizer asymmetry as an explanation for the observed gap.Both models use the same 32K tokenizer and reserved special-token IDs.Source: §3.1 · pp. 3–4

3.2 What “matched” does and does not mean

• The comparison is not matched in parameter count, total pretraining tokens, or curriculum shape, so it is a natural experiment rather than an ablation.The larger, longer-trained, more-SFT-ed model is the one without the capability, leaving bootstrap composition as the leading explanation in the authors’ account.Source: §3.2 · p. 4

4.1 Single-phase pretraining mix

• The 600M’s realized mixture was approximately 65% code-and-technical text, 34% Spanish, 0.7% cyber-specific text, and 0.1% instruction-formatted SFT.Configured bin weights differed from realized token fractions because the small cyber and SFT bins exhausted early.Source: §4.1 · p. 4, Table 2 · p. 4
• Small-bin exhaustion meant the 600M’s only chat-template and tool-call demonstration exposure was approximately 0.1% realized in-mix SFT, interleaved with bulk pretraining rather than following it.It had no dedicated conversational or tool-SFT stage.Source: §4.1 · p. 4

4.2 Infrastructure and run configuration

• The 600M used a single NVIDIA L4 and a planned 240,000-step, approximately 15.7B-token budget under a cosine learning-rate schedule.The reported configuration used BF16 and an effective batch of 65,536 tokens per optimizer step.Source: §4.2 · p. 4

4.3 The run is frozen at 64% of schedule

• At 64% of schedule, the 600M run stopped permanently before its cosine schedule annealed, so its reported results describe an unannealed, SFT-less snapshot.The checkpoint contains approximately 10B of the planned 15.7B tokens; no post-hoc stage was applied.Source: §4.3 · p. 5

4.4 The comparison model: VectraYX-1B’s curriculum

• The 1B curriculum included approximately 6B tokens of dedicated tool-SFT, but its terminal approximately 56B tokens were overwhelmingly non-code, dominated by web and educational prose.The curriculum also included later vision-SFT replay of the tool-SFT corpus.Source: §4.4 · p. 5

5 Evaluation Methodology

• The evaluation ladder ranges from the B1–B5 automatic harness to fixed prompts, strict reproduction and generalization tests, and no-generation mechanism probes.The disagreement between levels is itself treated as a result.Source: §5 · p. 5

5.1 Level 1: the B1–B5 harness (lenient, inherited)

• B4 uses automatic substring or keyword matching and can credit a tool name in prose without verifying a well-formed tool-call JSON object.Harness results are single-seed unless stated otherwise.Source: §5.1 · p. 5, Table 3 · p. 6

5.2 Level 2: fixed-prompt qualitative battery

• Six fixed prompts probe factual recall, enumeration, arithmetic, a domain definition, and two tool-eliciting situations to reveal categorical behavior differences.The battery is intended to catch distinctions such as spontaneous tool-call emission versus none, not finely rank models.Source: §5.2 · p. 5

5.3 Level 3: verbatim reproduction, and its generalization variant

• The strict check replays real training examples with their system preamble and training-time forward path, requiring a well-formed tool-call block and distinguishing generalized arguments from exact copies.A paired battery of eight corpus-absent prompts tests call decisions, all five tools, and extraction of unseen entities; the Level-3 suite takes minutes on CPU but uses only six reproduction examples and eight novel prompts.Source: §5.3 · pp. 5–6

5.4 Level 4: mechanism probes (no generation)

• The first-token probe reads the probability of <|tool_call|> at assistant-turn onset, while embedding-drift checks compare token rows and late-layer attention against the warm-start parent.The drift statistics must be interpreted against BF16 precision: unchanged entries may reflect no gradient or updates below precision, and the probe can falsify but not prove localization.Source: §5.4 · p. 6

6.1 Lenient harness: the two models look interchangeable

• The lenient harness made the models look like near-twins, with B4 and B5 agreeing to within 0.01.The 1B nominally led on B3, 0.24 vs. 0.17.Source: §6.1 · p. 6

6.2 Strict diagnostic, before remediation: the two models do not overlap

• The 600M produced well-formed, generalizing calls on 6/6 training examples, while the pre-remediation 1B produced the structure on 0 of 4–6 examples per run.The 1B failure persisted across tested pre-remediation checkpoints.Source: §6.2 · p. 7
• The 600M’s six valid calls used contextually reasonable arguments that differed from their training examples and from one another.The outputs demonstrate schema use rather than a lookup table of copied answers.Source: §6.2 · p. 7
• At the assistant-turn start, the 1B assigned only 10−4–10−5 probability to <|tool_call|>, indicating essentially no prior for the format.This persisted despite a dedicated ≈6B-token tool-SFT phase and further SFT on the probe corpus.Source: §6.2 · p. 7

6.3 The B4 discrepancy is a harness artifact

• The 1B’s B4 = 0.650 is unreliable: keyword matching credited tool-adjacent prose without a structured call, while the 600M’s B4 = 0.660 is corroborated by strict diagnostics.For the 600M, the lenient and strict instruments agree; for the 1B, they diverge across the metric’s full range.Source: §6.3 · p. 7
• Keyword harnesses fail open, so the authors treat verbatim reproduction as a mandatory gate before quoting structured-output benchmarks in this series.The paper identifies this as a second documented instance of the failure class, after Nano’s B2 [17].Source: §6.3 · p. 7

6.4 Qualitative battery: fluency without instruction-following, in two different registers

• The 600M stayed fluent and topical but did not answer four simple questions; it alone among the three tested checkpoints spontaneously emitted a well-formed tool call.Its topical output included Peruvian geography and credential-theft themes.Source: §6.4 · p. 7
• The two pre-remediation 1B checkpoints produced largely incoherent output, with the post-vision checkpoint worst and the earlier phase-3 checkpoint only marginally more fluent.The phase-3 checkpoint remained unable to answer or call a tool.Source: §6.4 · pp. 7–8
• The qualitative battery used the GGUF/Ollama path, which had a known RoPE tensor-layout export bug; export-bypassing strict diagnostics reproduce the same separation.Source: §6.4 · p. 8

6.5 Diagnosis-informed repair of the 1B

• Recipe B’s recorded 0/8 is unmeasured under the strict protocol because its harness omitted the training system preamble and silently deleted special tokens.The authors therefore treat recipe B as unmeasured, not as a failure.Source: §6.5 · p. 8
• Recipe C was rerun for 2,202 steps from a surviving sibling checkpoint, and every strict-diagnostic result for the repaired 1B uses that reproduced checkpoint.The historical checkpoint and its warm-start parent were lost; its figures are not load-bearing.Source: §6.5 · p. 8
• The repair’s training-example result alone could reflect rote memorization, so the novel-prompt generalization battery determines whether the calls demonstrate capability.Recipe C exactly reproduced four training completions, which the protocol does not count as capability on its own.Source: §6.5 · p. 8
• The reproduced repair emitted a well-formed call on all seven prompts requiring one and correctly suppressed the call on the eighth, achieving 8/8 structural correctness.The historical run scored 7/8 on both axes across the same eight prompts.Source: §6.5 · p. 8
• The repair handled unseen prompts requiring five-way tool selection and extracted entities absent from the 2,801-example corpus, supporting a generalizing tool-call policy.The recipe’s comparison does not isolate corpus diversity, learning rate, embedding unfreezing, or warm-start point as the effective ingredient.Source: §6.5 · p. 9
• The repair fixed tool calling, not instruction-following generally; the available comparison also cannot determine which recipe ingredient did the work.The authors report that the recipe succeeded from two sibling starting checkpoints, weakening the case for a warm-start-specific explanation.Source: §6.5 · p. 9

6.6 Locating the repair: routing, not representation

• The tied embedding table serves as both input embeddings and the unembedding matrix, so the drift check also tests the geometric direction the network must target to emit the token.Source: §6.6 · p. 9
• The <|tool_call|> row barely moved during repair, while attention projections moved measurably, supporting a change in routing rather than token representation.The row’s cosine similarity was 0.99999583; 97.7% of the embedding table’s entries were bit-identical, while sampled attention projections showed greater drift.Source: §6.6 · p. 9
• The table’s near-inertness reflects bf16 rounding: updates below half an ulp are discarded when weights are rounded back to storage precision.At the median absolute weight, the half-ulp threshold was 6.1×10−5; the per-step update at LR = 1×10−5 was roughly six times smaller.Source: §6.6 · p. 9
• The authors conclude that the repair taught network layers to route hidden states toward an already usable tool-call direction in the right contexts.The trigger representation was not measurably changed by the recipe.Source: §6.6 · p. 10
• The embedding result is not a two-arm ablation: the measured recipe left the table effectively frozen, so it cannot establish that unfreezing embeddings is generally inert.A higher learning rate or longer run could produce updates that clear the storage-precision threshold.Source: §6.6 · p. 10

Phase 2

• The 1B’s mean first-non-whitespace tool-call probability fell from 0.432 at step 36K to 5.9×10^-7 by step 48K, then stayed near the floor through phase 2.The decline was smooth across the densified 36K–48K window, within one probe regime.Source: §6.7 · p. 11, Table 7 · p. 12

Phase 2 and phase 3

• After step 38K, the 1B produced 0/5 well-formed calls at every phase-2 and phase-3 checkpoint, before its trigger-token probability reached the floor.Generation was already gone while the trigger token retained probability 3×10^-3 at 38K.Source: §6.7 · p. 11, Table 7 · p. 12

Phase 2

• Phase 2’s 36K emissions were byte-exact training copies, so the erased prior was memorized, not evidence of generalizing capability.The earliest archived checkpoint is 36K, leaving the prior’s formation unobserved.Source: §6.7 · p. 11

Phase 3

• In phase 3, the 1B’s trigger probability fell 24× after the tool-SFT mixture weight was cut from 0.20 to 0.16, then ended at 8.8×10^-5 with no calls emitted.The mixture was restored to 0.20 after step 10K, but the prior did not recover.Source: §6.7 · p. 11, Table 7 · p. 12

Harness and probe

• At step 14K, B4 peaked at 0.085 while the probe reached its phase-3 low of 5.7×10^-5; the model emitted 0/5 calls, so B4 did not measure actual tool calling here.The checkpoint-tracking B4 series is used only for within-series comparisons, not absolute scores.Source: §6.7 · pp. 11–12, Table 7 · p. 12

600M

• The 600M’s trigger prior had already risen before its first archived checkpoint, so the sampled trajectory shows saturation rather than emergence.The trajectory prediction is confirmed only at the endpoint.Source: Table 6 · p. 10, §6.7 · p. 11

Corpus prompts

• Across 269 corpus rows, valid-call emission was 0.959 for the repaired 1B, 0.926 for the 600M, and 0.100 for the pre-remediation 1B.Among well-formed calls, exact-copy rates were 0.988 for the repaired 1B and 0.235 for the 600M; the repaired model largely recited its training completions.Source: §6.8 · p. 12

Battery and scoring

• The unseen-entity battery used 238 prompts with novel entities and 166 call requests, scoring call decision, format, tool, argument shape, and entity or command-family correctness.The headline requires the right tool, argument shape, and novel entity; bash_exec instead requires a matching command family.Source: §6.8 · p. 12, Table 8 · p. 13

Component axes

• The 600M led in tool choice (0.776 vs. 0.717) and argument shape (0.855 vs. 0.819), while the repaired 1B led in unseen-entity extraction (0.667 vs. 0.560).Corpus-prompt recitation and out-of-distribution slot-filling therefore distinguish different behaviors.Source: §6.8 · pp. 12–13

Headline result

• The repaired 1B passed 0.536 of unseen-entity prompts against the 600M’s 0.428 (exact McNemar p=0.0039).On the paired items, 27 passed only for the repaired 1B versus 9 only for the 600M.Source: §6.8 · p. 12, Table 8 · p. 13

Template families

• No template-family ordering held across models: the 600M led on cisa_kev_check, while the repaired 1B led by more than 2× on bash_exec and terse_control.Both were near ceiling on nvd_get_cve and near floor on nvd_search; individual family gaps are imprecise at about 28 items per cell.Source: §6.8 · p. 13

Search terms

• Both models struggled with nvd_search because they emitted only the head noun, not the full multi-word search term; relaxed substring scoring raised headline scores to 0.470 and 0.590.Those values are for the 600M and repaired 1B, respectively; the ordering and paired conclusion were unchanged.Source: §6.8 · p. 13

Register

• On terse prompts, the 600M fell from 0.505 to 0.200, while the repaired 1B remained nearly flat at 0.532 → 0.500.The terse subset re-asked the same fixed-schema tools, varying register rather than tool set.Source: §6.8 · p. 13

Suppression and language

• On near-miss prompts, the 600M over-triggered on 0.913 and the repaired 1B on 0.783; the English–Spanish split is confounded by template and composition differences.The battery does not translate and balance the same templates across languages.Source: §6.8 · p. 13

Design

• The factorial held parent, step count, and batch constant while crossing narrow versus diverse corpora with low versus high learning rates; five runs included a duplicate diverse/high-rate seed.Runs used 2,202 steps; the narrow corpus had 12,801 examples and the diverse corpus 23,502.Source: §6.9 · p. 14

Scope of comparison

• The factorial’s five runs used fp16 on a T4, so their results are internally comparable but not bit-comparable with the bf16 recipe-C reproduction.The precision change matters because the earlier mechanism analysis concerned updates underflowing bf16 storage.Source: §6.9 · p. 14

Verbatim reproduction

• All five arms reached near-ceiling verbatim reproduction, so that in-distribution check did not distinguish their generalization performance.The subsequent comparisons use the 238-prompt battery.Source: §6.9 · p. 14

Seed variation

• The two diverse/high-rate seeds had identical headline pass rates, 0.6024, yet disagreed on six items and differed by +0.0241 on tool correctness.Matching aggregate rates therefore did not establish a stable cell-level result.Source: §6.9 · p. 14

Factorial findings

• At high learning rate, corpus diversity made no measurable difference; at low learning rate, raising the rate improved tool correctness in the narrow corpus, while the diverse-corpus contrast was inconsistent across seeds.The authors do not claim a learning-rate effect for the diverse corpus.Source: §6.9 · pp. 15–16

Interpretation

• The authors describe the ingredients as substitutable rather than additive, but do not estimate an interaction because three of four cells had only one completed run.They explicitly do not claim that the factorial identifies one dominant ingredient.Source: §6.9 · pp. 15–16

Suppression and seed sensitivity

• The apparent low-rate suppression gain for the diverse corpus was not replicated: one narrow-corpus seed differed from another by +0.181, as much as the corpus contrast.The suppression advantage remains a hypothesis; only one extra narrow/low-LR seed was tested, at 40% of training.Source: §6.9 · pp. 15–16

Format installation

• The narrow/low-LR arm installed the format, with well-formed calls on 0.994 of call items and a headline pass of 0.542.The result supports format installation by all repair configurations, not equivalent generalization across them.Source: §6.9 · p. 15

Setup

• Both arms started from the 600M step-154K checkpoint and shared 1,200 steps, a 1.5×10^-5 learning rate, and one L4 GPU; the distilled arm added about 4,443 teacher-generated records.The records were grounded in NVD CVE records and public Sigma rules.Source: §6.10 · p. 16

Distillation result

• The distilled-data arm showed no measurable effect versus baseline SFT: both passed the same fraction of call items, and no axis differed significantly.The paired headline-difference interval was [−0.036, +0.036].Source: Table 12 · p. 16, §6.10 · p. 16

Limitation

• Only one seed and a small sample were tested, so a small distillation effect cannot be excluded, though the data exclude a large one.The largest movements favored the distilled arm but were far from significance.Source: §6.10 · p. 16

Call and suppression behavior

• Against the untuned 600M, both SFT arms passed more call items (0.548 vs. 0.428) but eliminated its pure-prompt no-call answering; the baseline SFT called on 70 of 72 no-call prompts.The suppression score requires a non-degenerate answer, and some passing answers were factually wrong.Source: §6.10 · p. 16

7.1 Thesis: composition decides default vs. intervention

• Every tested repair configuration installed the call format, but suppression differences remain uncertain; the 600M emits by default while the 1B required a targeted repair.The factorial’s low-learning-rate suppression contrast was matched by seed variation, so the authors retain it as a hypothesis rather than a finding.Source: §7.1 · pp. 16–17, §7.4 · p. 19

7.2 Why would code-heavy pretraining install the default?

• Code-heavy data may favor structured output because format tokens and their local contexts are frequent in code, where syntactic constraints also recur [4].The discussion presents this as an explanation for the 600M’s default behavior, not as a demonstrated causal result.Source: §7.2 · p. 17
• The 600M’s default is an unconditional base rate, whereas the repaired 1B builds routing toward an unmoved trigger-token representation but transfers that gate imperfectly.The 600M calls on the pinned suppression example across checkpoints; the repaired 1B selects correctly among five tools on novel prompts, while suppression remains weaker.Source: §7.2 · pp. 17–18

7.3 Practical implications

• The paper recommends first-token telemetry, strict structural checks, and novel-prompt tests; its own harness once missed well-formed calls because decoding dropped special tokens.Disabling special-token dropping restored 8/8, and a related exporter issue contributed to repeated false emission findings.Source: §7.3 · p. 18

7.4 Proposed follow-up experiments

• Scaled tests extended verbatim checks to 269 examples and the battery to 238 prompts, finding that the repaired 1B beat the 600M on unseen-entity prompts and that both struggled with suppression.The held-out battery covers five trained tools with single calls, not unseen tools or multi-call sequences; the preamble-strength sweep remains open.Source: §7.4 · p. 19
• The factorial used four corpus-by-learning-rate combinations from one parent, but three cells had only one completed run and the T4 precision regime differed from the repair reproduction.The planned low-learning-rate seed replicates remain unfinished; the authors describe the factorial as five checkpoints, not a causal estimate.Source: §7.4 · p. 19
• The factorial found no measurable corpus effect at high learning rate, while the low-rate suppression advantage for diverse data remains unreplicated and seed-sensitive.The low-rate narrow-cell replicate at step 880 suppressed as well as the diverse arm, so the authors do not claim a dominant ingredient.Source: §7.4 · p. 19
• The composition claim still needs a matched-scale, fixed-token-count ablation; the trajectory sweep confirmed phase 2 erased an inherited 1B prior but left structured-syntax perplexity untested.The proposed composition experiment varies code fraction at fixed scale, while the remaining trajectory probe would test ordinary structured text beyond the trigger token.Source: §7.4 · pp. 18–19

• The 600M is frozen at 64% of its planned schedule, without cosine annealing, so its reported results describe an incomplete snapshot rather than a finished model.Training stopped at step 154,000 of 240,000 and cannot be resumed; the authors say this likely caps its quality.Source: §8 · p. 20
• The paper claims structured tool-call emission only; neither model has been shown to answer ordinary questions usefully or to provide a usable assistant experience.The 600M lacks dedicated conversational SFT, while the repaired 1B’s non-degenerate no-call answers were not checked for correctness.Source: §8 · p. 20
• The matched pair is not a controlled experiment: parameter count, pretraining tokens, curriculum, hardware, and composition differ, leaving composition a leading hypothesis rather than a demonstrated cause.The matched properties are architecture family, tokenizer, and token layout; the proposed causal composition ablation has not been run.Source: §8 · p. 20
• Initial strict diagnostics were small-n and used constructed prompts; the later battery enlarged testing, but still excludes unseen tools, multi-call sequences, and varying system-preamble strength.The final battery targets five trained tools with one call per item, and its no-call-answer check does not assess answer correctness.Source: §8 · p. 20
• The harness scores are single-seed and inherit risks from an export path with a prior RoPE bug; the 1B’s command-generation lead is also unaudited.The authors note that B3 may misestimate either model in either direction, just as they discredit the same harness’s B4 for the 1B.Source: §8 · p. 21
• The generalization battery has scorer and sampling caveats: fixing its argument check lowered pass rates, and duplicate suppression prompts leave 53 distinct items rather than 72.Its anchor prompts were reworded, and the soft-miss axis covers bash_exec alone with 30–31 items under a harder standard than other tool families.Source: §8 · pp. 21–22
• The repair is a reproduction because its original checkpoint and parent were lost, and the factorial’s ingredient conclusions remain limited by incomplete replication and a changed numerical regime.The historical results are retained for comparison but are load-bearing for nothing; low-learning-rate replication remains unfinished.Source: §8 · pp. 20–22

• The 600M passed verbatim reproduction on 6/6 examples, while the 1B failed on 0/4–6 before repair despite near-equal lenient scores.A diagnosis-informed repair then achieved generalizing five-way tool selection with novel-argument extraction on 8/8 prompts.Source: §9 · pp. 22–23
• The repair changed surrounding routing rather than the trigger-token representation: 97.7% of the embedding table remained bit-identical to its parent.The recipe used a diverse assistant-mode corpus, 5× learning rate, and about 3.3 GPU-hours.Source: §9 · p. 23
• The authors offer a minutes-of-CPU diagnostic ladder and conclude that composition relates to whether capability arrives by default or must be engineered, while acknowledged limitations constrain the claim.The limitations include confounded models, small-n diagnostics, and a reproduced rather than original repair artifact.Source: §9 · p. 23