PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantization Thresholds Replicate, Failure Modes Do Not: A Three-Model Study of Agentic Tool Use in Polish from 8-bit to 2-bit

Jakub Prejzner

arXiv:2609.32042v1cs.CL

TL;DR

Research on quantization of agentic tool use has been poorly characterized outside English. This paper measures Polish-interface tool use across multiple models and precisions, finding that the collapse threshold replicates while most other findings are model-specific.

  • Problem

    How quantization affects agentic tool use and whether findings transfer across models are poorly characterized, with nearly all existing evidence concerning English.

    Source: §1 · p. 1

  • Method

    PolAgentBench measures Polish prompts with English tool schemas across three models and six precisions using a 67-task main suite and an arithmetic ladder.

    Source: §1 · p. 2

  • Results

    The collapse threshold between 3-bit and 2-bit replicates across the three models, while the failure modes and other findings do not generalize uniformly.

    Source: §10 · p. 19, §12 · pp. 21–22

  • Takeaways & Limitations

    The findings support avoiding below-3-bit quantization for agentic work while cautioning against generalizing other behavior from a single model.

    Source: §10 · p. 19

    PLLuM failed the pre-specified easy-tier sanity gate, limiting its contribution to within-model contrasts and its failure profile.

    Source: §11 · p. 20

Abstract

from arXiv · show

We ask how GGUF quantization affects agentic tool use in Polish and whether the effects generalize across models. We introduce PolAgentBench, a deterministic benchmark with Polish prompts and English tool schemas: a 67-task main suite (15 adversarial probes, 52 hard-tier tasks) and a 46-task arithmetic isolation ladder. Three models span two axes of variation: Bielik-11B-v3.0 and its pruned, distilled child Bielik-Minitron-7B-v3.0 isolate model compression, and Llama-PLLuM-8B adds a change of pretraining family. Each is measured at six precisions, Q8_0 to Q2_K. Only the collapse threshold replicates. (1) All three models fall off a cliff between 3-bit and 2-bit (11B 0.716 to 0.045, 7B 0.463 to 0.149, PLLuM 0.224 to 0.015; paired McNemar p < 0.001 in each), across a fourfold capability spread and both axes. (2) Failure modes do not replicate: at 2-bit the 7B fails long (median 9.1k tokens, 4 steps) while the 11B mostly answers at the first step with a confabulated final answer (37 of 64 failures); PLLuM fails on content across precisions (71.8-92.0% of steps parse). (3) On the arithmetic ladder the unscaffolded rung is a floor, left standing by a rerun that states the no-tool rule; four explicit calls lift the 8-bit 11B from 1/10 to 9/10 and the 7B from 0/10 to 7/10 after format-only failures with the gold value are forgiven, an exploratory effect with eight distinct baseline inputs that does not survive multiplicity correction, while the order-trap arm separates the models at 8-bit (11B 6/6, 7B 0/6). (4) The Polish-versus-English gap is associated with degradation or with task family. We document four artifacts that shaped our conclusions (rounding-hostile gold values, strict answer typing, a no-tool rule the prompt never stated, priority-ordered failure labels), report affected results in strict and corrected form, and release the benchmark, trajectories and commit-stamped artifacts.

1 Introduction

The paper asks how quantization affects Polish agentic tool use and whether findings transfer across models, addressing a gap in evidence centered on English and static tasks. It introduces PolAgentBench and compares models across quantization and model-family variation, while documenting methodological hazards that affected its conclusions.

1 Introduction

  • The study addresses how quantization affects tool-using models and whether findings generalize, questions poorly characterized for non-English use.Small open-weight models are commonly deployed in 4-bit or 3-bit GGUF, while tool use increasingly involves reading outputs and chaining calls.Source: §1 · pp. 1–2
  • Polish prompts paired with English tool schemas require semantic translation while producing structured output, including normalization of inflected place names and numerals.Examples include converting locative Łodzi to nominative Łódź and parsing osiemdziesięciu as 80.Source: §1 · p. 2
  • The authors leave their original hypothesis about earlier agentic degradation untested because they did not measure general quality.The identical suite across three models instead separates compression within a family from a change of family; exactly one of four candidate findings replicates.Source: §1 · p. 2
  • PolAgentBench measures Polish-interface agentic tool use across three models and six precisions, with a 67-task main suite and a 46-task arithmetic ladder.The models vary through compression within a family and a change of pretraining family; the benchmark also includes two four-call arms and a length-by-language design.Source: §1 · p. 2
  • Four methodological hazards changed the authors’ conclusions, and affected ladder results are reported in strict and corrected forms.The hazards include rounding artifacts, strict answer typing, a no-tool rule omitted from the prompt, and a priority-ordered failure taxonomy.Source: §1 · p. 3

2 Related Work

  • Prior quantization evaluations cover static tasks or English, while this study extends measurement to agentic tasks, a non-English interface, and multiple models.Related agentic benchmarks are predominantly English and emphasize single-call correctness; TinyLLM covers small models but not quantization curves or non-English interfaces.Source: §2 · p. 3
  • The work builds on Polish model studies of Bielik and PLLuM, while earlier low-bit Bielik work measured general quality on one model rather than cross-model agentic trajectories.Bielik-Minitron-7B is derived from Bielik 11B through structured pruning and logit distillation; PLLuM is a separate Polish effort on an external lineage.Source: §2 · p. 3

3 PolAgentBench

PolAgentBench uses a deterministic weather-tool environment to test Polish prompts with English schemas across adversarial probes, multi-step chains, and an arithmetic ladder. Its design also documents scoring, decoding, and failure-label constraints that qualify how results should be interpreted.

3.1 Environment

  • Every task uses a deterministic weather environment with five pure-function tools over a static 52-city database and a hand-built neighbor graph.There is no randomness, clock, or I/O, so tool outputs are reproducible and run-to-run variation is localized to decoding.Source: §3.1 · p. 4

3.2 Task tiers

  • The 52-task hard tier includes 40 chains of 2–6 steps, while 15 easy-tier tasks probe adversarial failure modes such as inflection, hallucinated tool results, and over-calling.Order-trap chains distinguish following returned graph order from substituting a plausible-sounding answer.Source: §3.2 · p. 4

3.3 The arithmetic isolation ladder

  • The extended arithmetic ladder has 46 task slots: ten each for L0, L1, L2, and L3N, plus six L3T tasks.L0 supplies numbers directly, L1 and L2 require one and two calls, and the four-call arms either retain or remove the order trap; L0 initially omitted the no-tool prohibition.Source: §3.3 · p. 4
  • L0 and its rerun L0e each contain eight distinct inputs, while the six-task L3T arm reuses three pilot trap routes because the graph permits only nine genuinely distinct routes.The L3N construction uses separate conversion calls for the same gold value, a design choice intended to weaken rather than create the tested gradient.Source: §3.3 · pp. 4–5

3.4 Matched twins

  • Ten hard-tier chains are matched twins that share prompts, tools, and call sequences but differ in whether the oracle also requires the correct final number.The design isolates value correctness from orchestration correctness; three pairs also share inputs with pilot-ladder tasks.Source: §3.4 · p. 5

3.5 Length and language, deconfounded

  • The hard-tier design fills all four combinations of chain length and interface language to remove a prior confound between language and task length.Graph-free tools enable long English-only chains, while short Polish-prompt/English-schema chains fill the corresponding cells.Source: §3.5 · p. 5

3.6 Oracle, determinism, and three artifacts

  • Exact matching initially penalized legitimate rounded computations in 14 of 96 arithmetic failure slots, so the authors rebuilt gold values to be invariant to intermediate rounding.The pre-correction run placed the apparent collapse at 4-bit to 3-bit; on corrected data that contrast was not significant on any tier, and the cliff sat one step lower.Source: §3.6 · pp. 5–6
  • The answer-typing sensitivity correction forgives only format-tagged failures whose mistyped answer contains the gold value, and the 7B corrected rate is an upper bound.The no-tool rule was absent from L0 prompts even though the oracle enforced it; the rerun stated the prohibition, but the three pilot L0 tasks in the main suite were not rerun.Source: §3.6 · p. 6

4 Experimental Setup

The evaluation spans three models across six GGUF precisions, with the Bielik pair isolating compression within a family and PLLuM adding a pretraining-family change. PLLuM failed a pre-specified 8-bit sanity gate, so its results are used for within-model contrasts rather than between-model capability comparisons.

Models and quantizations

  • The three models were evaluated at six GGUF precisions, from Q8_0 through Q2_K.The Bielik-7B's Q3_K_M and Q2_K files were requantized from Q8_0 because the publisher release stopped at Q4_K_M.Source: §4 · pp. 8–9
  • The Bielik-7B is a pruned, distilled child of the Bielik-11B, while PLLuM adds a Polish-focused model from a different pretraining family at comparable scale.The Bielik pair shares architecture family, training corpus, and tokenizer; their comparison isolates model compression as a package, not parameter count alone.Source: §4 · p. 9

PLLuM provenance and scope

  • PLLuM scored 0.400 (6/15) at 8-bit, below the pre-specified 0.5 easy-tier gate, and was therefore excluded from between-model capability comparisons.Its parse rate and lack of chat-template artifacts supported reading it as a weak tool-use model on this suite, while prompt compatibility remained a limitation.Source: §4 · p. 9

Evaluation protocol

  • Inference used temperature 0, seed 42, an 8192-token context window, and task-specific step budgets; paired McNemar tests and task-resampling bootstrap intervals supported the comparisons.Independent oracle re-evaluation reproduced every reported pass count for the Bielik main-suite runs (24 of 24).Source: §4 · pp. 9–10

5 Finding 1: The collapse threshold replicates in all three models

All three models decline sharply between Q3_K_M and Q2_K despite different capabilities and model lineages. Other curve features differ, so the shared collapse threshold is the main replicated result.

Shared collapse threshold

  • All three models fall sharply between Q3_K_M and Q2_K in the 67-task all-task pass-rate table.The table reports results by model and precision, with repair off and on.Source: Table 2 · p. 10, §5 · pp. 10–11
  • The threshold recurs across a fourfold 8-bit capability range and both compression-within-family and pretraining-family variation.The authors qualify the conclusion to Polish-focused models in the 7B to 11B range.Source: §5 · p. 11

Other curve patterns

  • The 7B curve is a statistical plateau through Q3_K_M, while PLLuM moves within a low 0.104–0.224 band before the cliff.The 11B's Q4_K_M dip is described rather than claimed as a robust effect because its significance changes under twin exclusion and correction.Source: §5 · p. 11
  • At Q2_K the Bielik-11B scores 0.045 against the Bielik-7B's 0.149, reversing their ordering at other precisions.The authors describe this as greater headroom for the 11B across the usable range but less graceful degradation at the extreme.Source: §5 · p. 11

6 Finding 2: Failure modes do not replicate; three models, three signatures

Failure signatures diverge across the models, even though their aggregate curves share a collapse threshold. In the Bielik pair, opposite trajectory patterns at 2-bit and a distinct 7B protocol defect at 8-bit show why failure labels and mechanisms do not transfer directly.

6 Finding 2: Failure modes do not replicate; three models, three signatures

  • Failure labels are priority-sensitive: the 7B's 8-bit-to-5-bit plateau and PLLuM's labels are indicative, while the 7B's 3-bit fixation result is better determined.PLLuM's 130 of 140 failures are labeled genuine arithmetic under the as-run priority, but 90 also carry fixation tags; the table covers 25 arithmetic-graded tasks.Source: Table 3 · p. 11, §6 · p. 12

6.1 Two opposite collapses in the Bielik family, visible in trajectory cost

  • At 2-bit the 7B fails over a median 9,139 tokens and 4 steps, while the 11B fails over 1,506.5 tokens and one step.The 11B has a parsed final answer in 50 of 64 failures; 37 end at the first step with a well-formed answer unsupported by any tool output.Source: §6.1 · p. 12, Table 4 · p. 13
  • The Bielik models' opposite 2-bit failure patterns support the warning that a failure-mode taxonomy fitted to one model should not be assumed to transfer.Their shared lineage and tokenizer make the contrast within this pair meaningful, although pruning, distillation, and post-training cannot be separated as causes.Source: §6.1 · pp. 12–13

6.2 Envelope collapse: one defect behind the 7B’s 8-bit protocol failures

  • At 8-bit, the 7B sometimes drops the required call_tool envelope on the third distinct tool, causing an unknown-action rejection and a wrong-tool-order verdict.On three-tool EN_EN chains it passes 0 of 6 slots; one-shot envelope repair converts 5 of 6 to passes, but the effective sample is three prompts and no significance test is run.Source: §6.2 · pp. 13–14

7 Finding 3 (exploratory): The unscaffolded floor, scaffolding, and the order trap

The ladder leaves both Bieliks at an unscaffolded floor, even after an explicit no-tool instruction; four-call scaffolding lifts both at 8-bit, but the effect is exploratory. The order-trap arm separates the models, while matched twins show that value penalties are sparse and partly confounded by divergent greedy decodes.

7 Finding 3 (exploratory): The unscaffolded floor, scaffolding, and the order trap

  • L0 scores 0.00 in all eight Bielik runs, while explicit no-tool rerun L0e scores 0.10 in one cell and 0.00 in the other three.The original prompt omitted the no-tool rule; L0e adds it, and the control reproduced the original outputs and verdicts.Source: Table 5 · p. 14, §7 · pp. 14–15
  • At 8-bit, four-call scaffolding raises the corrected 11B rate from 1/10 to 9/10 and the 7B from 0/10 to 7/10, but none of twelve contrasts survives Holm correction.The ladder has ten slots but eight distinct baseline inputs; the effect is reported as exploratory.Source: Table 6 · p. 15, §7 · p. 16
  • At 8-bit, the order-trap arm yields 6/6 passes for the 11B and 0/6 for the 7B; both models score 0/6 at 4-bit and 2-bit.The trap-free arm succeeds for both at 8-bit, with corrected rates of 0.90 for the 11B and 0.70 for the 7B.Source: §7 · p. 16
  • The 11B beats its 8-bit result on light scaffolding at 3-bit, solving L1 and L2 at 10/10 each versus 1/10 and 4/10, but the ordering reverses on the trap-free four-call arm.The 8-bit 11B remains ahead overall on the main suite, and the inversion is a property of the measured greedy configuration, not a stable ranking.Source: §7 · pp. 16–17

7.1 Matched twins: the value penalty, and its robustness to the typing artifact

  • Across ten matched twin pairs, the coinciding-trajectory value penalty is small and one-directional, while nine pairs differ because greedy decodes diverge.The absolute twin levels remain exposed to the typing artifact; the artifact cannot create a structure-pass/value-fail discordance on coinciding trajectories.Source: §7.1 · p. 17

8 Finding 4: The language gap tracks degradation or task family

The Polish-English pass-rate pattern varies by model and chain length, but the design compares task families as well as languages. The observations therefore do not isolate an interface contribution from task family.

8 Finding 4: The language gap tracks degradation or task family

  • For the 7B, Polish leads on short chains at 8-bit (0.78 against 0.33), while English leads on long chains (0.08 against 0.56).The long Polish-English cell contains 11 of 13 tasks in the four-tool order-trap family.Source: §8 · p. 17
  • For the 11B, Polish matches or exceeds English in the long-chain bucket at 8-bit, 6-bit, 5-bit, and 3-bit; English leads only in Q4_K_M.The cells are matched on length band, not exact length or task family, so they do not isolate an interface contribution from task family.Source: §8 · pp. 17–18
  • The four cells match chain-length bands, not exact lengths or task families, so the observed language association cannot be isolated from task family.Every long Polish chain uses the neighbour graph, while no long English chain does.Source: §8 · pp. 17–18

9 Under sampling, collapsed runs carry more format failures

A three-seed sampling probe of the 11B is bimodal: two runs stay near greedy performance and one collapses at both tested precisions. The collapsed run is associated with format-tagged failures, but the small probe does not establish a mechanism or stable sampling rate.

9 Under sampling, collapsed runs carry more format failures

  • At temperature 0.7, two 11B seeds land near the greedy result and one collapses at both tested precisions.The probe covers two configurations and three seeds; the authors treat it as an observation, not a finding.Source: §9 · pp. 18–19, Table 9 · p. 19
  • Forgiving every format-tagged failure puts collapsed seed 1 at 0.776 for Q8_0 and 0.881 for Q3_K_M, but these are ceilings, not recovered pass rates.The forgiveness also credits failures with wrong final answers or other content tags.Source: §9 · p. 18
  • Across length-language cells, the 7B’s long Polish-English cell is a floor, while the 11B’s Polish rate matches or exceeds English at healthy precisions except the 4-bit dip.The section’s discussion links the observed association to degradation in the parent and an unsolved structural family in the compressed child, without isolating language.Source: Table 8 · p. 19, §10 · p. 19
  • At Q3_K_M, the collapsed seed repeats most: seed 1 has 9.8% repetition, versus 2.5% and 3.6% for the other two seeds, so repetition is not excluded there.Source: §9 · p. 19
  • The authors retain temperature 0 because the sampling probe shows that a pass rate at temperature is not a stable single number for this model.The seed is supplied per step, restarting each step’s sampling stream; this uniform harness detail does not explain the collapse.Source: §9 · p. 19
  • The collapsed seed is associated with format-tagged failures, but forgiveness bounds do not establish that format degrades before reasoning.The authors connect this pattern to format failures also implicated in the 7B’s envelope collapse and the typing artifact.Source: §9 · p. 19

10 Discussion

  • The shared deployment rule is to avoid running these models below 3 bits for agentic work, but other observed behavior should not be generalized from one model.The discussion distinguishes the shared cliff location from model-specific failure modes, scaffolding, and language patterns.Source: §10 · p. 19
  • For the 7B, enabling repair lifts the full-suite 8-bit rate from 0.448 to 0.567, while repair changes nothing for the 11B at 8-bit or 3-bit.For PLLuM, repair adds at most 0.045 anywhere; the authors use repair as a diagnostic of failure type.Source: §10 · p. 20

11 Limitations •

The limitations constrain attribution and generalization: the Bielik comparison bundles compression changes, PLLuM is a weak family-axis anchor, and several analyses rely on small or imperfectly reproducible samples. The authors also disclose oracle artifacts, synthetic conditions, and unresolved anomalies.

11 Limitations •

  • The Bielik pair isolates compression as a package, not scale: pruning, distillation, post-training, and parameter count cannot be separated.The components of compression are not individually separable in this comparison.Source: §11 · p. 20
  • PLLuM is a weak family-axis anchor: it failed the pre-specified easy-tier sanity gate at 8-bit (0.400 versus a 0.5 threshold).Its contribution is limited to within-model contrasts and its failure profile; prompt-style disadvantage cannot be ruled out.Source: §11 · p. 20
  • Quantization provenance differs across models, and the decisive oracle was not modified after the typing artifact was found.Affected results are reported at strict and corrected rates; main-suite twin absolute levels were not re-audited for the artifact as a whole.Source: §11 · p. 20
  • Failure labels are priority-dominated in specific cells, and the trigger for envelope collapse is not isolated from tool position or tool identity.No 2-bit claim rests on the failure labels.Source: §11 · pp. 20–21
  • The 67 main-suite task slots contain 54 distinct inputs, while the sampling probe covers only two configurations and three seeds.Twin contrasts use n=10 and were never individually significant.Source: §11 · p. 21
  • The ladder is underpowered for confirmatory inference after multiplicity correction: eight distinct baseline inputs yield a minimum p of 0.0078, above Holm’s 0.0042 threshold.The no-tool rerun corrected only four of fourteen ladder cells; some original L0 contrasts and pilot tasks remain as run.Source: §11 · p. 21
  • Greedy decoding was not fully reproducible across matched twins: raw outputs matched in 45/60 11B pairs, 28/60 7B pairs, and 27/60 PLLuM pairs.The grid therefore represents one draw from a decoder whose output may depend on more than the prompt.Source: §11 · p. 21
  • The benchmark uses a synthetic deterministic environment with no tool latency or failure variability.The 11B Q4_K_M dip and single-seed sampling collapse remain open anomalies without reported mechanisms.Source: §11 · p. 21

12 Conclusion

Across three models and six precisions, only the 3-bit-to-2-bit collapse threshold replicates; failure modes and arithmetic-ladder behavior differ. The paper also reports reproducibility details and identifies the scope of the easy-tier results.

12 Conclusion

  • The collapse threshold between 3-bit and 2-bit replicates across three models and a fourfold capability spread, but their failure modes do not.At 2-bit, the 7B fails at a median of 9.1k tokens over 4 steps, while the 11B often gives a confabulated answer at the first step; PLLuM fails on content while maintaining protocol fluency.Source: §12 · p. 21
  • The unscaffolded arithmetic rung remains a floor after the no-tool rule is stated, while forgiving format-only failures lifts both Bieliks at 8-bit in an exploratory result.The corrected 7B rate is an upper bound, and the effect does not survive multiplicity correction at this sample size; the four-call arms’ model split is associated with the order trap.Source: §12 · pp. 21–22
  • The interface gap is associated with degraded configurations or an unsolved task family, while four scoring artifacts shaped interpretation without changing every result in the same way.Affected ladder results are reported in strict and corrected form; the no-tool issue is also measured by a rerun, and priority-ordered failure labels are less determinate than they appear.Source: §12 · p. 22

Reproducibility

  • The evaluation used the 67-task main suite and 46-task ladder, with released trajectories, task definitions, outcome matrices, analysis scripts, and commit-stamped artifacts.Independent oracle re-evaluation reproduced all Bielik main-suite counts, and the release reports 204 unit tests passing at stamped commits.Source: Reproducibility · pp. 22–23

A Easy-tier per-task outcomes, Bielik pair

B Tier-resolved counts and the repair flag

Repair effects vary by model and tier, while failure-taxonomy counts depend on rule priority. Paired contrasts and bootstrap intervals characterize the quantization differences, with the 3-bit-to-2-bit cliff the most consistent result.

B Tier-resolved counts and the repair flag

  • Repair adds 7B hard-tier passes from 8-bit through 4-bit, but at 2-bit adds only easy-tier passes; 11B gains are confined to hard-tier tasks and PLLuM gains are mostly easy-tier.The 7B repair gains are +8, +7, +5, and +6 tasks from 8-bit through 4-bit, with no gain at 3-bit.Source: Table 11 · p. 24, Table 12 · p. 24

C Failure taxonomy audit

  • PLLuM failures are often multi-label: 90 of 140 combine wrong-final-answer and fixation tags, while 39 failures trigger only one rule branch.The counts are affected by priority ordering; reversing the priority would relabel 101 of 140 failures.Source: C Failure taxonomy audit · p. 25

D Paired McNemar contrasts, main suite

  • Every contrast against Q2_K is significant across the three models, while contrasts above the cliff are generally non-significant except for the 11B’s nominal Q4_K_M comparisons.Without twin tasks, all cliff contrasts remain significant; the 11B Q4_K_M contrast against Q3 is no longer supported by an interval excluding zero.Source: D Paired McNemar contrasts, main suite · p. 26, Table 14 · p. 26, Table 15 · p. 26

E Bootstrap intervals

  • Bootstrap differences on 47 non-twin tasks preserve the cliff’s sign and verdict, while the 11B Q4-versus-Q3 interval includes zero.For Q3 versus Q2, the paired differences are +0.340 for 7B, +0.617 for 11B, and +0.277 for PLLuM, each with an interval excluding zero.Source: E Bootstrap intervals · p. 27, Table 17 · p. 27, Table 18 · p. 28

F Token and step statistics, Bielik pair

The token-and-step table defines the scope of the Bielik-pair distributions, while the 7B’s long-chain outcomes differ by chain type. A worked order-trap example shows distinct 11B and 7B trajectories on the same task.

F Token and step statistics, Bielik pair

  • Token and step distributions are reported by outcome, with per-task budgets of 4 to 10 steps and 8 steps on 29 of 67 tasks.Token counts are comparable only within the Bielik pair because they share a tokenizer.Source: Table 19 · p. 28

G Length and language, per chain length, 7B

H Worked example: a 4-tool order-trap chain

  • On the worked order-trap task, the 11B passes in 6 steps by converting the two days separately, while the 7B uses all 8 steps without a final answer.The 11B’s longer route passes because the oracle accepts an ordered subsequence of required calls.Source: H Worked example: a 4-tool order-trap… · p. 29

I Envelope collapse, raw output

A 7B trajectory illustrates protocol-envelope collapse, while the arithmetic ladder analysis distinguishes strict scores from corrected rates. The correction can forgive format-only failures with gold values, but the 7B corrected rate is an upper bound.

I Envelope collapse, raw output

  • The 7B’s first two calls are well formed, but its envelope disappears at the third distinct tool; repair converts 5 of 6 affected task slots to passes.The example also contains an arithmetic error, and sibling prompts show both bare-JSON and wrong-type answer formats.Source: I Envelope collapse, raw output · p. 30

J Extended ladder detail and the typing artifact

  • Typing correction changes only four L3N cells: 11B Q8_0 rises from 0.20 to 0.90 and 7B Q8_0 from 0.00 to 0.70.No L0, L1, L2, or L3T cell changes at any precision.Source: J Extended ladder detail and the… · p. 30
  • Forgiven 7B trajectories can include rejected steps, so its corrected rate also forgives those steps and is therefore an upper bound.Correcting answer type alone would leave eight of the forgiven trajectories failed because the oracle rejects any trajectory with a rejected step.Source: J Extended ladder detail and the… · p. 30
  • At original L0, the 11B makes no tool calls at the reported precisions, while the 7B attempts collapsed-envelope calls; stating the no-tool rule in the rerun leaves the floor standing.The rerun’s control reproduced the original outputs, token counts, and verdicts for all 80 corresponding trajectories.Source: J Extended ladder detail and the… · p. 31
  • With eight distinct baseline inputs, no ladder contrast survives Holm correction; the smallest attainable exact p-value already exceeds the smallest-test threshold.L0 and L0e each have only eight distinct inputs because two pairs share the same prompt, tools, and budget.Source: J Extended ladder detail and the… · p. 32

K Sampling probe detail

Each step’s completion call receives the sampling seed, restarting the sampling stream at every step. Seed 1’s deficit is associated with format-tagged failures, but the forgiveness analyses do not establish that formatting degrades before reasoning.

K Sampling probe detail

  • Every step’s completion call receives the sampling seed, so its sampling stream restarts from the same seed; this is uniform across seeds.Source: K Sampling probe detail · p. 33
  • For seed 1, forgiving all format-tagged failures yields 0.776 at Q8_0 and 0.881 at Q3_K_M, while forgiving format-only failures yields 22/67 and 30/67.Allowing mixed failures but excluding wrong_final_answer yields 0.687 and 0.746; these ceilings associate the collapsed seed with format-tagged failures without showing that format degrades before reasoning.Source: K Sampling probe detail · p. 33
  • Under the same forgiveness, seed 1 is within the other seeds’ band at Q3_K_M but below both at Q8_0; stricter forgiveness leaves it below both bands at both precisions.At Q8_0, seeds 2 and 3 reach 0.910 and 0.955 under the same forgiveness, and 0.881 and 0.925 under stricter forgiveness.Source: K Sampling probe detail · p. 33
  • Repeated content does not explain the collapsed Q8_0 seed: seed 3 repeats most at 13.7% of steps and scores 48/67, while seed 1 repeats most at Q3_K_M.At Q3_K_M, seed 1 repeats at 9.8%, versus 2.5% and 3.6% for seeds 2 and 3.Source: K Sampling probe detail · p. 33

Related papers

Loading 2609.32042v1…