PaperlayerView exact revision on arXiv

Source-linked AI summary

Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference

Gaoyuan Du, Anam Nawaz Khan, Rex Zhou, Xiaoyang Liu, Deepayan Chakrabarti, Fnu Suya, Xueping Li

arXiv:2609.26621v1cs.LGcs.AI

TL;DR

BF16 and FP16 can produce different greedy outputs for the same model and prompt, despite repeatability within each format. The paper analyzes the role of top-two margins and evaluates selective FP32 lm_head recomputation; it reports higher exact agreement at low latency overhead, with benefits bounded by inference conditions.

  • Problem

    The same model, prompt, and greedy-decoding configuration can produce different outputs across numerical formats, and within-format determinism tools do not reconcile those outputs.

    Source: §1 · p. 2

  • Method

    The paper analyzes cross-precision error propagation and evaluates margin-gated FP32 recomputation of the lm_head.

    Source: §7.3 · p. 14, §7.6 · p. 15, §9 · p. 17

  • Results

    +22–36pp exact agreement on A10G is reported for gated FP32 lm_head recomputation, with less than 4% latency overhead.

    Source: §9 · p. 17

  • Takeaways & Limitations

    Cross-precision agreement gains are hardware-dependent, and the intervention is intended for low-batch greedy decoding rather than as a universal determinism guarantee.

    Source: §8.1 · p. 16, §8.4 · p. 16

    At batch sizes ≥8 and under end-to-end FP8, divergence becomes body-dominated and head-only repair is insufficient.

    Source: §8.4 · p. 16, §8.2 · p. 16, §8.1 · p. 16

Abstract

from arXiv · show

Greedy decoding from large language models is commonly treated as deterministic. We show it is not precision-invariant: the same model, prompt, and decoding algorithm produce different outputs in BF16 versus FP16 on identical hardware. Across our evaluations of six models (1.1B-7B parameters, four families; divergence additionally characterised at 12B) and three benchmarks, 49-100\% of prompts diverge; a single token flip often cascades into trajectory-level divergence. We develop an empirical error-propagation analysis and find that 22 layers of accumulated body error do not distinguish flipping from non-flipping steps; the outcome depends primarily on the top-two logit margin at the LM head relative to the directional perturbation between the top-two candidates. The analysis makes five testable predictions about intervention outcomes, including that applying more FP32 compute (broader scope) makes agreement worse. The experiments match all five predictions. The best-performing low-overhead intervention we evaluate, selective FP32 LM head recomputation, triggered only when the margin falls below a threshold, delivers +22-36 pp exact agreement on A10G (+12-21 pp on L4 and A100) at less than 4\% latency overhead in low-batch (batch size <=4) single-stream inference. We map the applicability boundary across six models and four batch sizes, and hypothesise that training-time precision stability is a determining factor. The method is a partial mitigation rather than a universal determinism guarantee: its benefit vanishes when body-originated error dominates, including at batch size >=8 and under end-to-end FP8 in our tests.

1 Introduction

The paper measures frequent output disagreement between BF16 and FP16 greedy decoding and argues that low top-two margins at the lm_head, rather than body-error magnitude alone, govern token flips. It tests margin-gated FP32 interventions and maps where their agreement gains apply.

1 Introduction

  • Cross-format replay can fail even when repeated runs are bit-identical within each precision, complicating auditing and comparisons of individual predictions.The paper also reports that BF16 Intervention C moves outputs closer to FP32, while FP16 alone remains closer than BF16+C.Source: §1 · p. 2
  • 49–100% of greedy generations differ between BF16 and FP16 across the evaluated models and benchmarks.The study also reports 59–70% divergence for TinyLlama-1.1B across GSM8K, HumanEval, and MBPP, and 82% for Qwen2.5-3B-Instruct.Source: §1 · p. 2
  • Token flips concentrate where the top-two margin is small relative to directional logit perturbation, with the lm_head acting as the dominant amplification stage.The analysis reports a per-logit perturbation scale σz ≈0.026 and convergence across four tracing and grafting experiments.Source: §1 · pp. 2–3
  • All five intervention predictions are confirmed, including that extending FP32 computation beyond the lm_head reduces agreement from 63% to 44%.The predictions include no effect from integer quantization or temperature sharpening and comparable outcomes from top-K and full-head recomputation.Source: §1 · p. 3
  • +36pp to 0pp is the reported range of intervention lift across evaluated models; effectiveness appears to track training-time precision stability, a stated hypothesis.The remedy survives FlashAttention-2 in the tested setting, while the lm_head-only scope vanishes at batch size ≥8.Source: §1 · p. 3
  • 24/59 divergent prompts end with different output lengths, illustrating how one token flip can cascade into trajectory-level divergence.The mean length difference is 34 tokens and the maximum is 236 tokens.Source: Figure 1 · p. 3

2 Related work

The paper distinguishes cross-precision output divergence from established within-format numerical nondeterminism and positions its work as a targeted characterization and mitigation. It also specifies how benchmark Pass@1 is measured.

2 Related work

  • Pass@1 denotes greedy-decoding final-answer correctness measured under bare prompts without a system prompt or chat template, with matched-pair McNemar testing.For HumanEval, the paper additionally reports execution-based pass@1 in an appendix.Source: §2 · p. 3
  • BF16/FP16 serving is standard, but its output-level consequences and targeted mitigation remain incompletely characterized.Prior work also addresses within-format nondeterminism from floating-point operations, batching, and kernel scheduling, which the paper distinguishes from cross-format divergence.Source: §2 · pp. 3–4
  • LayerCast reduces system-level numerical variation through layerwise FP32 computation, while this paper localizes a correctable component to the lm_head and uses selective recomputation.The paper reports +38% A10G latency for its global FP32-compute baseline versus less than 4% for the selective intervention; PrecisionDiff instead characterizes a sensitivity surface.Source: §2 · p. 4

3 Problem formulation

The formulation treats each autoregressive step as a precision-dependent logit computation followed by an argmax. A small gap between the leading logits can allow rounding differences to flip the selected token and alter later conditioning.

3 Problem formulation

  • At each step, the model computes logits from the prompt and prior tokens, then selects the token with the largest logit; logits depend on precision, tensor parallelism, and GPU architecture.The formulation names BF16, FP16, and FP32 as numerical precisions that can affect the logits.Source: §3 · p. 4
  • When the top-token gap is small enough for configuration-dependent rounding to flip the argmax, the selected tokens differ and autoregressive conditioning diverges.The resulting cascade length is defined as Lcascade = T − tdiv.Source: §3 · p. 4
  • The paper measures exact sequence agreement, first divergence step, trigger rate, and mean gap, plus semantic agreement for tasks with well-defined final answers.Semantic agreement compares extracted conclusions even when full reasoning differs.Source: §3 · p. 4

4 Inference-time interventions for cross-precision agreement

The interventions test whether divergence arises from comparison-level ties, inaccurate candidate logits, or error amplified at the lm_head. They selectively act at small-margin steps, with different correction scopes and computational costs.

4 Inference-time interventions for cross-precision agreement

  • All five mechanism predictions are confirmed, including zero effect from comparison-only changes, comparable top-K and full-head recomputation, and worse agreement from broader FP32 scope.The intervention family changes the final decoding step without retraining, modifying the transformer body, or adding persistent GPU memory.Source: §4 · pp. 4–5

4.4 Overhead and trade-off analysis

  • Selective gating triggers only when the margin falls below τ = 10−3, affecting 0.7–1.4% of steps; evaluated interventions add less than 4% latency.Supplementary temperature sharpening has zero effect, while consensus decoding achieves perfect agreement at approximately twice the compute budget.Source: §4 · p. 5, §4.4 · pp. 6–7

4.1 Intervention A: Integer logit quantization

  • Integer logit quantization maps values within τ to the same bin and breaks ties by choosing the smaller token index, but cannot reconcile logits that differ across precisions.The method deliberately discards sub-τ differences and adds one division and one rounding operation per vocabulary element.Source: §4.1 · pp. 5–6, §4.2 · p. 6

4.2 Intervention B: Top-K FP32 recomputation with integer comparison

  • For top-K recomputation, the method recalculates candidate logits in FP32 before integer comparison; K = 2 suffices to match the full-vocabulary version in tested gated cases.The tested K values from 2 through the full vocabulary produce identical EAR, and the K = 8 implementation is reported as 4000× cheaper per trigger than full-head recomputation.Source: §4.2 · p. 6

4.3 Intervention C: Full-vocabulary FP32 lm_head recomputation

  • Full-vocabulary FP32 recomputation upcasts the hidden-state projection’s weights and activations on the fly, avoiding persistent FP32 weight copies.The lm_head projects the hidden state over the full vocabulary and is identified as the dominant source of cross-precision logit perturbation.Source: §4.3 · p. 6

5 Experimental setup

The experiments compare BF16 and FP16 greedy decoding across public math and code benchmarks, using TinyLlama as the main testbed and additional models for sensitivity analysis. They also define the precision comparisons and baselines used to measure divergence.

5 Experimental setup

  • The primary evaluation uses GSM8K for math and HumanEval and MBPP for code.The setup uses 8-shot chain-of-thought on GSM8K, 0-shot HumanEval, and 3-shot MBPP.Source: §5 · p. 7
  • TinyLlama-1.1B-Chat is the main testbed; sensitivity analysis compares reasoning-tuned DeepSeek-R1 models with instruction-tuned Qwen2.5 models.TinyLlama was selected for fast iteration and full ablations on one GPU.Source: §5 · p. 7
  • BF16 and FP16 greedy decoding are compared on the same A10G GPU, with fixed seeds and a 256-token generation limit.The setup also checks that deterministic-algorithm flags do not remove cross-precision divergence.Source: §5 · p. 7
  • BF16 versus FP16 is used as a controlled comparison because both are common 16-bit inference formats but allocate exponent and mantissa bits differently.Transfer to other format pairs remains future work.Source: §5 · p. 7
  • The baselines include standard BF16/FP16 decoding, an FP32 oracle, global FP32 compute, and ungated FP32 recomputation.The FP32 oracle is defined as eliminating precision-induced divergence.Source: §5 · p. 7
  • 30–41% exact agreement (EAR) is reported across the TinyLlama-1.1B divergence-spectrum table’s datasets.The table reports 95% Wilson score intervals.Source: Table 2 · p. 8

6 Results

The results show that BF16 and FP16 can diverge despite deterministic kernels, across task types and at different points in generation. Selective FP32 LM-head recomputation improves agreement, but its benefits depend on the source of error and model or workload conditions.

6.1 Sanity check: kernel-level determinism is not the cause

  • 100% within-precision self-consistency coexists with a 41% BF16-vs-FP16 EAR that persists under both loose and strict determinism settings.The divergence also persists when the same BF16 weights use native BF16 versus FP32 compute, implicating arithmetic rounding rather than kernel dispatch.Source: §6.1 · p. 8

6.2 Divergence is substantial across tasks

  • 64–70% of code-generation prompts and 59% of math-reasoning prompts produce different BF16 and FP16 outputs.For non-Qwen models, median first divergence occurs at 34–50 tokens; Qwen2.5-7B-Instruct diverges at the first token in all tested prompts.Source: §6.2 · p. 8

6.3 Intervention comparison: a controlled ablation

  • The FP32 oracle yields 100% agreement, while global FP32 upcasting reaches 88–89% but doubles memory footprint.Gated recomputation achieves 55–63% agreement at less than 4% overhead; ungated recomputation reaches only 54–60% at about 2.5× latency.Source: §6.3 · pp. 8–9
  • +22pp on GSM8K and +25pp on HumanEval and MBPP: gated FP32 recomputation improves exact agreement with less than 4% overhead.Interventions B and C deliver identical gains; Table 3 identifies gated LM-head recomputation as the best tested efficiency–agreement trade-off.Source: §6.3 · p. 9, Table 3 · p. 9
  • 22 of 30 triggered GSM8K prompts are reconciled by gated C, while it does not trigger on 29 prompts whose divergence involves large accumulated error.The 22 reconciliations account for the +22pp exact-agreement lift; Table 4 stratifies the per-prompt cases.Source: §6.3 · p. 9, Table 4 · p. 10

6.4 Semantic agreement and gating sensitivity

  • On GSM8K, semantic agreement is 58% versus 41% exact token agreement at baseline, and Intervention C raises semantic agreement to 72%.On Qwen2.5-3B-Instruct, 19% of GSM8K prompts flip correctness across precisions even though aggregate accuracy differs by only 1pp (27% vs. 28%).Source: §6.4 · pp. 10–11

6.5 Model sensitivity: reasoning vs. instruction tuning

  • +6.5pp with less than 4% overhead for gated C on DeepSeek-R1-1.5B, compared with +2.5pp and 2.5× latency for ungated recomputation.At matched parameter scales, reasoning-tuned models show 10–15pp lower exact agreement and 1.7× higher trigger rates than instruction-tuned counterparts.Source: §6.4 · p. 11, §6.5 · p. 11

7 Mechanism: an empirical analysis of error propagation

The analysis finds that flips are associated with small top-two logit margins, not body-layer error magnitude alone. It explains when selective FP32 LM-head recomputation can reconcile paths and where that repair stops helping.

7.1 Error propagation: from machine epsilon to logit perturbation

  • A flip occurs when a competitor’s differential logit perturbation exceeds its original logit gap, while a practical RMS-scale trigger is heuristic rather than an exact bound.The heuristic classifies flip versus no-flip correctly on 88–89% of steps across two model families.Source: §7.1 · pp. 11–12

7.2 Why FP32 recomputation works

  • At gated steps, deterministic FP32 LM-head recomputation can reconcile paths when their hidden states agree; it improves EAR by 22pp, reconciling 22 of 30 prompts entering the gate.That is a 73% success rate within scope.Source: §7.2 · pp. 12–13

7.3 When lm_head-only scope is sufficient, and when it is not

  • The estimated LM-head contribution is 97.6% of per-logit perturbation; positive lift occurs when the cross-arm hidden-state gap is well below the margin scale, but is negligible when it exceeds that scale.The four tested configurations span nearly three orders of magnitude in ∆h/σz; lift magnitude itself is not monotone in ∆h.Source: §7.3 · pp. 13–14

7.4 Why ungated recomputation hurts

  • Ungated FP32 recomputation can introduce a third trajectory at otherwise safe steps, producing +2.5pp versus +6.5pp for gated recomputation on DeepSeek-R1-1.5B.The authors describe gating as mechanistically necessary.Source: §7.4 · p. 15

7.5 Empirical validation: layer-level divergence tracing

  • Across 22 traced layers, body-layer L2 error is indistinguishable between diverged and agreed groups, while their top-two margins differ by 150× (0.039 vs. 5.733).The passage reports four experiments converging on the LM head as the source of the flip.Source: §7.1 · p. 12, §7.5 · p. 15

7.6 Summary

  • EAR rises from 0.41 to 0.82 when only matmul compute is upgraded to FP32; extending FP32 beyond the LM head reduces EAR from 63% to 44–52%.The summary attributes about 87% of the reachable EAR gap to BF16 arithmetic and associates a residual of about 12pp with weight truncation.Source: §7.6 · p. 15

8 Discussion

The intervention is intended for low-batch greedy decoding and can improve exact agreement, but it does not guarantee bit-exact replay. Its usefulness depends on the model, platform, and inference setting.

8.1 Practical implications

  • At batch sizes ≤4, the intervention is intended for low-batch greedy decoding; at batch sizes ≥8, head-only repair provides no lift by itself in the tested settings.Results remain positive with both tested attention kernels, while lift varies substantially by GPU.Source: §8.1 · p. 16

8.2 What does partial agreement provide?

  • On A10G, exact whole-sequence agreement rises from 30–41% to 55–63% at <4% latency overhead, but the intervention does not provide bit-exact reproduction.On Qwen2.5-3B HumanEval, execution outcomes agree on 97.6% of problems while token sequences agree on 26.2%; on GSM8K, 19% of prompts show a cross-precision correctness flip.Source: §8.2 · p. 16

8.3 Practitioner decision procedure

  • CV(TD), computed from one BF16 run, is associated with intervention EAR lift across five models (Spearman ρ = 0.90; Pearson r = 0.84), but remains an in-sample heuristic requiring target-hardware recalibration.The passage says it is not a validated cross-model predictor.Source: §8.3 · p. 16

8.4 Limitations

  • The intervention is not evaluated for sampling-based decoding or at 12B and 70B+ full-comparison scales, and it is insufficient by itself at batch sizes ≥8 and under end-to-end FP8.The authors also report that changing only LM-head arithmetic cannot remove the measured ∼12pp BF16/FP16 weight-storage gap.Source: §8.4 · p. 16

9 Conclusion

Across the tested models and benchmarks, BF16 and FP16 greedy decoding diverge on 49–100% of prompts, while gated FP32 LM-head recomputation narrows agreement gaps in low-batch inference. The conclusion also stresses that the intervention is not a guarantee of identical outputs.

9 Conclusion

  • 49–100% of prompts diverge between BF16 and FP16, while gated FP32 LM-head recomputation improves exact agreement by +22–36pp on A10G at <4% latency overhead.Selected L4 and A100 replications yield +12–21pp; the method is evaluated up to 7B, with divergence additionally characterised at 12B.Source: §9 · p. 17

Broader impact

  • Cross-precision reproducibility matters for exact replay, audit trails, and per-example evaluation, but reproducibility does not establish correctness or replace domain-specific validation.The intervention changes numerical consistency rather than model capability and is not a substitute for accuracy or safety validation.Source: Broader impact · p. 17

Appendix overview

  • The appendix groups supporting material into analysis, mechanism evidence, cross-model robustness, production-stack robustness, ablations, and statistical validation and reproducibility.The listed groups cover appendices I–XI.Source: Appendix overview · p. 20

I Error-propagation analysis: derivations and assumptions

  • The error-propagation model is heuristic except for Proposition 1, relies on simplifying assumptions, and is empirically validated rather than proven for arbitrary weight configurations.The text notes that adversarial weight configurations can contradict its asymptotic rates.Source: I Error-propagation analysis:… · p. 20

I.1 Setup and notation

  • The setup defines an L-layer autoregressive transformer with hidden dimension d and vocabulary size |V| for analysis at decoding step t.Source: §I.1 · p. 20

I.2 Error accumulation through the transformer body

The analysis models accumulated transformer-body error, then relates token flips to the top-two logit margin and its perturbation. It motivates selective FP32 LM-head recomputation, which reconciles some—but not all—reachable divergences.

I.2 Error accumulation through the transformer body

  • Under stated residual-connection, bounded-rounding-error, and no-cancellation assumptions, the body-error estimate simplifies to linear accumulation when per-layer error terms are small.For TinyLlama-1.1B, measured BF16-vs-FP16 hidden-state L2 grew from 0.002 at layer 0 to 0.92 at layer 22, consistent with that regime.Source: §I.2 · pp. 20–21

I.3 Logit perturbation and the divergence condition

  • A token can flip when a competitor’s differential perturbation exceeds its original logit gap; empirically, flips concentrate at small top-two margins relative to perturbation scale.The estimated per-logit perturbation is σz ≈0.026; the reported median margin is 0.039 for diverged prompts versus 5.733 for agreed prompts, a 150× gap between groups in the headline trace.Source: §I.3 · p. 21

I.4 Intervention C: error reduction at the lm_head

  • FP32 LM-head recomputation reduces head-specific rounding error, but residual body-originated differences remain when the two precision paths have different hidden states.For LM-head-originated divergence with shared hidden states, the residual difference vanishes; the maximum achievable lift is limited by the fraction of prompts diverging only at the head.Source: §I.4 · p. 21
  • 22 of 30 TinyLlama prompts with LM-head margin-triggered divergence were reconciled by Intervention C, a 73% success rate within that scope and a 22pp achieved lift.The analysis predicted a maximum 30% lift for this reachable set.Source: §I.4 · p. 21

I.5 Extended derivation and edge cases for the lm_head dominance estimate

The extended analysis separates body and LM-head contributions to logit perturbation and identifies settings where body-originated error makes an LM-head-only intervention insufficient.

I.5 Extended derivation and edge cases for the lm_head dominance estimate

  • The extended estimate decomposes per-logit perturbation into projected transformer-body error and LM-head accumulation under a standard architecture with KV caching.The derivation describes a body term from accumulated layer error projected onto a logit direction, alongside rounding in the LM-head multiply-accumulate.Source: §I.5 · p. 22
  • During prefill, attention reductions across prompt positions add body-error terms that can approach or exceed the LM-head contribution for long prompts.For TinyLlama, the analysis estimates this boundary at prompt lengths around 850 tokens, where LM-head-only scope becomes insufficient.Source: §I.5 · p. 22

I.6 Scope boundary: when Intervention C fails

  • When FP16 casting of weights trained in BF16 yields unstable intermediate activations, body error dominates and FP32 LM-head recomputation may provide no lift.The text predicts and reports 0pp lift on DS-R1-Qwen-7B.Source: §I.6 · p. 22
  • At batch size 8, attention-reduction order creates a third trajectory that head recomputation does not reconcile, and the intervention’s lift vanishes.The reported bs=1-vs-bs=8 hidden-state difference is 0.0127; the measured lift at bs=8 is 0pp.Source: §I.6 · pp. 22–23
  • Different BF16 and FP16 stored weights leave an approximately 12pp agreement gap even when both arms compute in FP32.The reported EAR is 0.88 rather than 1.00; full agreement in this comparison requires a shared storage format.Source: §I.6 · p. 23

I.7 Intervention comparison from the error model

This section evaluates whether the error model’s intervention predictions hold and where cross-precision flips arise. It also examines how Intervention C’s gains vary across tested models and batch sizes, informing deployment choices without implying a guaranteed family-level taxonomy.

I.7 Intervention comparison from the error model

  • Quantization and temperature sharpening have zero effect, consensus decoding removes format-level divergence at 2× compute, and K = 2 suffices for identical EAR across tested K values.Quantization’s bin width is 10^-3 versus cross-precision logit difference σz ≈ 0.026; sharpening preserves the margin-to-perturbation ratio.Source: §I.7 · p. 23

II Mechanism evidence: four causal experiments

  • Four independent experiments converge on a late causal locus: divergence arises across the last transformer layer, final RMSNorm, and lm_head, where an FP32 correction can affect both paths’ logits.The experiments triangulate the claim through hidden-state error tracing, layer grafting, KV-cache grafting, and per-layer top-2 traceback.Source: §I.7 · p. 23, II Mechanism evidence: four causal… · pp. 23–24

(i) Unconditional per-layer hidden-state trace

  • All 100 first-token measurements agree despite hidden-state L2 error growing from 0.0021 at layer 0 to 0.9341 at layer 21 and logit-level error reaching 4.6275.Final RMSNorm and lm_head amplify error, but the implied per-logit perturbation ≈0.026 remains below the mean top-two margin of 0.70.Source: (i) Unconditional per-layer… · p. 24

(ii) Causal attribution via layer grafting

  • Body-layer grafts fix only ∼36% of flips, while Layer 21 and final RMSNorm grafts fix 65% and 71%, respectively, localizing causal responsibility to the final stages before the lm_head.Even embedding grafts fix only 29.4%; the table reports the same decisive jump from body layers to Layer 21 and final RMSNorm.Source: (ii) Causal attribution via layer… · pp. 24–25, Table 8 · p. 25

(iii) Causal role of the KV cache

  • KV-cache grafts convert 47.5% of flips from BF16 to FP16 argmax and 35.6% in the reverse direction, showing the cache is a partial, asymmetric cause.Combining graft results, the authors attribute the residual ∼21% of “kept BF16” outcomes under RMSNorm graft to the BF16 lm_head matmul; graft percentages are not additive.Source: (iii) Causal role of the KV cache · pp. 25–26

(iv) Per-layer top-2 traceback via logit lens

  • BF16 and FP16 top-1 tokens agree at body layers for 93–100% of prompts, and top-2 sets agree at least 93% throughout; divergence is a one-shot ranking reversal at the lm_head.Diverged prompts decide late: their margin shrinks from 0.33 to 0.04, whereas the Agreed group’s grows from 2.06 to 5.73; the pattern replicates across GSM8K, HumanEval, and MBPP.Source: (iv) Per-layer top-2 traceback via… · pp. 26–28, Table 10 · p. 27

III Divergence-conditioned trace on Qwen2.5-3B-Instruct

  • On Qwen2.5-3B-Instruct, top-two margins differ 90× between groups (0.088 vs. 7.925), with 54% vs. 0% flip rates, while body L2 remains indistinguishable.The authors note that the two model families support the core mechanism while body-layer patterns vary by model.Source: III Divergence-conditioned trace on… · p. 28

IV Cross-model intervention replication

  • Intervention C lifts exact agreement by +36pp on Llama-3.2-3B, +22pp on TinyLlama-1.1B, +3pp on Qwen2.5-3B, +8pp on Mistral-7B, +10pp on OLMoE, and 0pp on DS-R1-Qwen-7B.The 7B comparison shows scale alone does not explain the split: Mistral gains while DS-R1-Qwen does not; the authors hypothesise effectiveness tracks training-time precision stability.Source: IV Cross-model intervention replication · pp. 28–30, Table 12 · p. 29
  • +20 to +36pp EAR in tested 1–3B Llama-family models, +3 to +10pp in Qwen2.5-3B, Mistral-7B, and OLMoE, and no standalone gain in BF16-saturated Qwen variants.These observed tiers describe the tested models, not a guaranteed family-level taxonomy.Source: IV Cross-model intervention replication · p. 30
  • Intervention C gains were +17pp at batch size 1, +7–8pp at batch sizes 2 and 4, and 0pp at batch size 8.Composing C with global FP32 compute recovered +5pp at batch size 8 in the tested setup.Source: IV Cross-model intervention replication · p. 30
  • Practitioners choosing between C and body-scope alternatives such as LayerCast should first measure the target model’s divergence rate and margin distribution on the target platform.Source: IV Cross-model intervention replication · p. 30

V 7B-scale scope boundary: DeepSeek-R1-Distill-Qwen-7B

On DeepSeek-R1-Distill-Qwen-7B, Intervention C triggers but does not restore BF16–FP16 agreement. Divergence usually begins within the first few tokens, where body-originated error is outside the lm_head-only remedy’s correctable regime.

V 7B-scale scope boundary: DeepSeek-R1-Distill-Qwen-7B

  • 0% EAR across all four arms: none of the tested interventions recovers agreement on DeepSeek-R1-Distill-Qwen-7B.The intervention’s ∼5% latency increase confirms that it triggers rather than being silently skipped.Source: Table 13 · p. 31, V 7B-scale scope boundary:… · p. 31
  • Median first-divergence step is 6, and 86% of prompts diverge within the first 9 tokens.This early window is where accumulated transformer-body error and KV-cache drift put the flips beyond lm_head-only correction.Source: Table 14 · p. 31, V 7B-scale scope boundary:… · p. 31
  • The FP32 lm_head trigger fires, but by then the autoregressive path has already entered a precision-specific KV-cache branch.All 100 prompts fall on this cache-locked branch in the experiment.Source: V 7B-scale scope boundary:… · p. 31
  • The authors hypothesise that training-time precision exposure contributes to this pattern, but do not verify that causal mechanism.They recommend pinning serving precision for 7B-scale BF16-saturated models and leave body-scope alternatives unevaluated on this model.Source: V 7B-scale scope boundary:… · pp. 31–32

VI Quality preservation: does Intervention C change task accuracy?

Across the tested model and prompt-format conditions, Intervention C improves cross-precision agreement without a statistically significant measured task-accuracy change. On longer MATH-500 chains, baseline divergence remains similar to GSM8K and the intervention retains positive point-estimate lifts.

VI Quality preservation: does Intervention C change task accuracy?

  • The native chat-template replication leaves both McNemar tests non-significant, while FP16 has zero Pass@1 flips under Intervention C.Llama-3.2-3B BF16 baseline Pass@1 rises from 24% to 66% with the native chat template.Source: VI Quality preservation: does… · p. 32, Table 16 · p. 33
  • No tested model–precision cell shows a statistically significant accuracy change under Intervention C; the four-model evidence supports quality preservation in tested conditions, not proof of zero effect.Mistral-7B point estimates drop by 3–5pp but are non-significant; at n=100, the data cannot distinguish noise from a small negative effect.Source: VI Quality preservation: does… · pp. 32–33

VII Long-reasoning sanity check: does divergence grow with chain length?

  • On MATH-500, baseline divergence is 44% for TinyLlama and 30% for Llama-3.2-3B, within 3pp of GSM8K despite 3–4× longer chains.Intervention C’s point-estimate lifts are +20pp and +26pp, respectively; n=50 intervals are descriptive, not paired lift intervals.Source: VII Long-reasoning sanity check: does… · pp. 34–35

X Statistical validation and reproducibility

Scaled replications support the agreement gains, while the paper reports reproducibility settings and a directional analysis separating body-error projection from the top-two margin. Intervention C also moves BF16 outputs toward FP32 sequence-level behavior in the tested models.

X Statistical validation and reproducibility

  • At n=300 GSM8K, n=164 HumanEval, and n=257 MBPP, scaled lifts are +16.4pp, +28.1pp, and +27.2pp, each with McNemar p < 0.001.Per-cell confidence-interval half-widths fall to ±5.5–7.4pp; the TinyLlama +22pp lift has a 90% bootstrap CI of [+11, +33]pp.Source: X Statistical validation and… · p. 35

XI Reproducibility statement

  • The experiments use greedy decoding with fixed prompts and deterministic flags; EAR compares complete token-id sequences.The primary setup uses an A10G, with L4 and A100 cross-hardware replications.Source: XI Reproducibility statement · pp. 35–36

XII Directional body-error analysis

  • Body projection is statistically equivalent between Diverged and Agreed steps, while their top-two margins differ by two to three orders of magnitude.The directional decision rule classifies flips correctly on 88% of TinyLlama steps and 89% of Qwen2.5-3B steps; it is an analysis tool, not Intervention C’s gate.Source: XII Directional body-error analysis · pp. 36–37, Table 20 · p. 36

XIII FP32 fidelity of Intervention C

  • Intervention C raises FP32 sequence agreement by +21pp on TinyLlama and +19pp on Qwen2.5-3B.FP16 alone has higher FP32 fidelity than BF16+C, so the claim is narrower: within a fixed BF16 deployment, the intervention moves outputs toward the FP32 reference.Source: XIII FP32 fidelity of Intervention C · p. 37, Table 21 · p. 37

XIV Cross-architecture replication: L4 (Ada), A100 (Ampere), and T4 (Turing)

The intervention yields positive agreement lifts on tested GPU architectures, but their magnitude depends on hardware. FP8 tests show that threshold scaling matters when the body is held fixed, whereas end-to-end FP8 and execution-based outcomes expose important limits and distinctions between token and task agreement.

XIV Cross-architecture replication: L4 (Ada), A100 (Ampere), and T4 (Turing)

XV FP8 replication and threshold scaling

XVI Execution-based correctness (HumanEval)

XVII Cross-task EAR on Qwen2.5-3B-Instruct

  • Qwen2.5-3B shows 60–63% divergence on the two code benchmarks, comparable to 75% GSM8K divergence under the same protocol.The appendix notes that bare-prompt GSM8K figures are not directly comparable to the main-text chat-template figures.Source: Table 29 · p. 41, XVII Cross-task EAR on… · p. 42

Related papers

Loading 2609.26621v1…