Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2

1 Introduction

• The paper asks when and where SR improves transformer inference accuracy over RN at low fixed precision, since operation length and network location affect error accumulation and propagation.An MLP down-projection reduces over 3072 terms, compared with 64 for an attention-head score.Source: §1 · p. 2
• VPSR extends PRISM to arbitrary virtual precision, with a proof of correctness and reported speeds up to 70× faster than Verificarlo’s Monte Carlo Arithmetic random rounding.The paper holds the numerical format fixed and varies only the rounding rule.Source: §1 · pp. 2–3
• The analysis combines a vector Bienaymé–Chebyshev bound for MLP projections with a second-order decomposition of cross-entropy change into signed shift, drift curvature, and variance penalty.Together, the analyses characterize site-level trade-offs between SR and RN.Source: §1 · p. 3
• At t = 6 in the MLP, SR limits perplexity degradation to 15% over the full-precision reference, versus 121% for RN.The MLP gap widens as more down-projections are affected; at t = 8, SR comes within 0.3% of the reference.Source: §1 · p. 3

2.1 Stochastic rounding

• SR is unbiased on average, while RN has the smaller worst-case error per operation; probabilistic bounds replace RN’s length-n dot-product error factor nu with one of order √nu under SR.RN errors can align, and small addends may be absorbed by a large accumulator.Source: §2.1 · pp. 3–4

2.2 Emulating stochastic rounding in software

• EFTs recover an operation’s exact result as a rounded value plus residual, but standard EFT methods are limited to hardware precision; VPSR decouples virtual precision without extended-precision overhead.QPyTorch’s uninstrumented high-precision matrix-multiplication accumulator does not capture error accumulation along intermediate reduction chains.Source: §2.2 · p. 4

2.3 Low-precision transformer inference

• Quantization methods map values onto a grid while holding the rounding rule fixed; this paper fixes the mapping and varies the rule, studying interactions with network propagation and downstream loss.Prior discussion describes a prevailing preference for RN in forward passes, while the paper argues that SR’s performance differs across network sites.Source: §2.3 · p. 5

2.4 The architecture under study

• A GPT-2 block applies an attention residual and then a separately normalized MLP residual; its final linear head produces vocabulary logits scored by cross-entropy.Three matrix multiplications reduce over dmodel, while the MLP down-projection reduces over dff; attention also reduces over head dimension and sequence length.Source: §2.4 · p. 6

3 Variable-precision stochastic rounding (VPSR)

• VPSR truncates the hardware-rounded value to a virtual precision t, then combines the virtual truncation error ρ and hardware residual τ to determine the stochastic rounding decision.This decouples the rounding decision from the hardware format.Source: §3 · pp. 6–7
• The algorithm compares the signed distance ρ + τ with a uniform threshold on [0, ulp_t); using r random bits approximates that continuous draw and introduces a bounded, negligible bias.Fixing the threshold at half an ulp_t instead gives untied round-to-nearest at precision t.Source: §3 · p. 8
• VPSR handles fragile two-term sign evaluation, binade-boundary spacing, and subnormal virtual grids through a hardware-arithmetic comparison, predecessor-exponent selection, and common scaling.For subnormal grids, the algorithm scales quantities by S = 2^64 before comparing in the normal range.Source: §3 · pp. 8–9

3.1 Exactness of the rounding decision

• Under the theorem’s hypotheses, the hardware floating-point evaluation preserves the sign of the rounding decision boundary, establishing exact decision evaluation.The hypotheses hold in Algorithm 1, including the standard round-to-nearest bound on the hardware residual.Source: §3.1 · p. 9

4 PRISM: vectorized probabilistic rounding at variable precision

• The Verificarlo pass replaces floating-point instructions with PRISM calls that compute an exact EFT pair and round the result at virtual precision.Each call processes all lanes in one kernel invocation; SR and RN share the kernel, with RN fixing z = 1/2.Source: §4 · p. 10, §4.1 · p. 10, Figure 3 · p. 10

4.1 Instrumentation and kernels

• The kernels use lane masks and selection for data-dependent conditions, keeping control flow branch-free across lanes.PRISM kernels are written against Google Highway and compile to x86, Arm, RISC-V, and WebAssembly targets.Source: §4.1 · pp. 10–11

4.2 Run-time configuration

• Precision, rounding mode, and generator state are runtime settings, and PyTorch module hooks change them around each target module without recompilation.Worker threads reload cached settings after the process-wide configuration epoch advances.Source: §4.2 · p. 11

4.3 Performance

• PRISM SR incurs ×8.5–32.1 native-time overhead, versus ×88–597 for MCA RR; PRISM is up to ×4.7 faster for binary32 and ×70.5 for binary64 operations.The comparison covers elementwise addition, multiplication, division, and FMA over arrays of 65,536 values.Source: §4.3 · p. 11, Table 1 · p. 11

5.1 Forward error analysis of linear projection layers

• The GELU backward-error framework lacks a uniform guarantee near its local minimum, so the analysis isolates affine projections whose errors directly perturb the residual stream.At z* ≈ −0.752, GELU’s derivative vanishes while its value is nonzero, making the framework’s condition-number term diverge nearby.Source: §5.1 · pp. 12–13
• For fixed α in the first-order regime, affine-projection error is O(√nu) under SR versus O(nu) under deterministic RN; SR obtains the √n factor through the Bienaymé–Chebyshev variance bound.At t = 7, the RN bound is 2.43 × 10^10, while the SR vector BC bound is 4.73; both bounds are worst-case over inputs. The vector bound bypasses the union bound, keeping its confidence multiplier 1/√α independent of output dimension m.Source: §5.1 · pp. 14–15

5.2 Error propagation and loss decomposition at the output softmax

• Expected excess loss separates signed drift, drift curvature, and variance; the Fisher-weighted penalty charges non-uniform logit changes, while uniform shifts leave softmax unchanged.RN has zero operational variance on a fixed token, so its perturbation is entirely drift; SR avoids drift terms but pays a variance penalty.Source: §5.2 · pp. 17–18

5.3 Empirical validation of the drift–variance trade-off

• At the MLP down-projection, RN shows larger per-run perturbations than SR, but most RN drift is a uniform logit shift; at the head, SR perturbations are larger and non-uniform.Predominantly positive MLP inputs can bias and accumulate RN errors, whereas zero-centered head inputs promote cancellation; head SR is unbiased but incurs higher local variance.Source: §5.3 · pp. 18–19
• At the head, SR’s variance penalty accounts for 100–105% of its excess loss over t = 6, . . . , 9, while its drift and curvature terms lie within 1.6 standard errors of zero.RN is deterministic, so its perturbation has no variance penalty; the head’s non-uniform fluctuations across the vocabulary cannot be ignored by softmax.Source: §5.3 · p. 19
• At the MLP down-projection, RN’s aligned errors produce a curvature penalty of 0.29 nats, while SR keeps curvature to 0.017 nats.The 3072-length inner product allows RN errors to align across terms; SR prevents this systematic alignment.Source: §5.3 · p. 19
• At t = 6, MLP SR noise has 6.23 logits RMS versus 4.27 at the head, yet its p-centered fluctuation yields a variance penalty of 0.083 nats, fifty times smaller than at the head.The MLP injects noise into a 768-dimensional hidden state rather than directly into all 50,257 logits, coupling the logits so the fluctuation is predominantly a uniform shift that softmax ignores.Source: §5.3 · pp. 19–20
• At t = 6, head SR excess loss is 4.40 nats versus RN’s 2.12, while MLP down-projection SR is 0.05 versus RN’s 0.24.Figure 6 compares the decomposition at both sites; the prose reports the corresponding t = 6 losses.Source: §5.3 · p. 19, Figure 6 · p. 21

6.1 Experimental setup

• DistilGPT-2 was evaluated on 1,020 WikiText-2 next-token positions, with target operation scopes compared at matched precision and five SR seeds per configuration.The pretrained checkpoint was used without fine-tuning, calibration, or quantization-aware adjustment.Source: §6.1 · pp. 21–22

6.2 Results

• Across uniform precision t=4–11, RN significantly outperformed SR; head-only reduction reproduced most of the gap, suggesting the head drives the global RN preference.At t=8, perplexity was 97.68 under RN and 139.27 ± 3.58 under SR.Source: §6.2 · p. 23, Figure 7 · p. 24
• At t=6, MLP perplexity was 71.97 ± 1.16 under SR versus 137.93 under RN, but head perplexity was 5054 ± 464 under SR versus 519.18 under RN.MLP perturbations reach logits predominantly as uniform shifts, whereas head noise changes differences between logits.Source: §6.2 · pp. 23–25, Figure 8 · p. 24
• The MLP down-projection (n=3072) had the largest SR advantage, while n=768 projections showed different preferences: attn_c_attn favored RN and attn_c_proj and mlp_c_fc favored SR at select precisions.The sweep does not isolate reduction length: the n=3072 site also differs in input distribution, weights, and downstream operations.Source: Figure 9 · p. 25, §6.2 · pp. 25–26
• Across mlp_c_proj block prefixes, SR significantly lowered perplexity at tested precisions except the first prefixes at t=6, with the advantage growing as more blocks were affected.The growing gap is consistent with RN drift shifting later activations; RN excess loss also exceeds the sum of single-block losses more than SR excess loss does. Separately reduced attention projections favor RN at t=4, while reducing the whole attention module favors SR.Source: §6.2 · pp. 23, 25, 26, Figure 10 · p. 27

6.3 Mixed rounding beats full RN at matched precision

• At identical per-site significand precisions, mixed rounding lowered perplexity 3.04%–28.17% versus all-RN across all three k values.The tested recipe used SR at the attention and MLP output projections; its gains grew as the MLP down-projection lost precision, and replacing RN with SR at those outputs compensated for removing one significand bit there.Source: §6.3 · pp. 26–28, Table 3 · p. 27

6.4 Limitations

• The evidence is limited to one six-block model and one text prefix; emulation retains the binary32 exponent range, so broader-task and native-hardware portability remain unestablished.Native-hardware transfer also depends on reduction order and arithmetic implementation.Source: §6.4 · p. 28

• At t = 6, all six MLP down-projections yield 6.0% degradation with SR versus 27% with RN (66.24 against 79.64 perplexity).Source: §7 · p. 28
• SR yields an 11.6× perplexity reduction at t = 5 in the MLP; mixed recipes improve perplexity 3.04%–28.17% over matched all-RN configurations.Source: §7 · p. 28
• At the MLP down-projection, RN drift has greater curvature cost than SR variance, which is small because propagated SR noise is predominantly a uniform vocabulary shift.At the language-model head, SR fluctuations are non-uniform and penalized directly by cross-entropy.Source: §7 · pp. 28–29
• The measured site ordering and composition effects need replication across texts, larger models, and broader tasks; the analysis does not establish a universal rounding assignment.The complete sweeps use one six-block model and one leading text prefix.Source: §7 · p. 29
• Evaluation pipelines, experiment scripts, and aggregated data are publicly available and archived on Zenodo.Source: §7 · p. 29

• Under Hypotheses 1–3, evaluating the rounding-decision boundary in floating point preserves the exact mathematical sign.Source: Appendix A · p. 29
• The proof sign-normalizes ρ, π, and τ so that the variables have nonnegative values under the stated cases.For ρ ≠ 0, the normalized ρ is |ρ|; for ρ = 0, the normalized τ is |τ|.Source: Appendix A · p. 29
• The proof rules out collapse to zero by bounding the normalized operands, applying monotonicity, and using Sterbenz’s Lemma to make the subtraction exact.Source: Appendix A · pp. 29–30

• The decomposition measurements use the DistilGPT-2 checkpoint and WikiText-2 positions, with full-precision reference loss L0 = 4.1354 nats and PPL0 = 62.51.Instrumented operations outside the lowered site use matching rounding at t = 24; the SR background contribution is reported as negligible.Source: Appendix B · p. 30
• The variance penalty is computed from coordinate variances and the variance of p⊤∆ without forming the full covariance matrix; SR reports mean ± 1 SE over 32 seeds, while RN is deterministic.The unbiased estimate of Q can be negative; the reported head estimates at t = 8 lie within two standard errors of zero.Source: Appendix B · p. 30, Table 4 · p. 31
• Table 4 gives the loss-decomposition terms in nats averaged over N = 1020 scored positions.Source: Appendix B · p. 30, Table 4 · p. 31