Calibrating the Checksum: An Empirical False-Positive Bound for One-Sided ABFT on bfloat16 GEMM, and What It Fails to Catch

• At bfloat16 and K=4096, the classical tau=Ku threshold is undefined because Ku≈32 exceeds the bound’s Ku≤0.01 condition.Source: §1 · p. 2
• The threshold is extrapolated from a Generalized Pareto fit to 7.99 × 10^7 clean per-row residuals at a stated per-GEMM false-positive rate of 10^-6.Source: §1 · p. 2
• The per-row-to-per-GEMM conversion’s independence assumption is checked empirically rather than asserted.Source: §1 · p. 2
• Corruption detection is characterized by which bfloat16 bit is flipped and how many bits are flipped at once.Source: §1 · p. 2
• Detectability did not vary with contraction length across a 16× range, falsifying the mechanism the study was designed to demonstrate.Source: §1 · p. 2
• The detector failed its detection bar, which was registered before any data existed.Source: §1 · p. 2
• The results characterize output-tensor perturbations, not fault incidence in real silicon.Source: §1 · p. 2

2.1 Silent data corruption

• Silent data corruption is a computation that completes incorrectly, with production reports from Google and Meta describing the problem at scale.Source: §2.1 · p. 2

2.2 Checksum ABFT

• Checksum ABFT checks an exact-arithmetic invariant; this paper uses a one-sided per-row check, while scalar reduction discards row-level information.Source: §2.2 · pp. 2–3

2.3 Thresholds from numerical bounds, and where they stop working

• At K=4096, bfloat16 gives Ku≈32, beyond the Ku≤0.01 condition behind the classical threshold, so it is undefined [Ma et al., 2025].Source: §2.3 · p. 3

2.4 Thresholds from model resilience

• Model-resilience thresholds target task degradation, whereas floating-point checksum thresholds must also avoid false positives; integer GEMM has no rounding-noise floor [Xue et al., 2023] [Xie et al., 2025].Source: §2.4 · p. 3

2.5 What is missing

• Prior work supplied no empirical bfloat16 checksum threshold calibrated to a stated per-GEMM false-positive rate, the gap this paper addresses.Source: §2.5 · pp. 3–4

2.6 Two negative results, from opposite directions

• Ma et al. [2025] reported checksum ABFT misses on 14 of 15 unhealthy production nodes; this study reaches a compatible negative conclusion through controlled bfloat16 injection.Source: §2.6 · p. 4

2.7 Fault injection and its abstraction level

• Algorithm-level injection does not reproduce instruction-level or micro-architectural effects, bounding what these results support.Source: §2.7 · p. 4

3.1 The residual statistic

• The one-sided detector compares per-row checksum vectors after float64 promotion, normalizing their difference with the nonnegative contraction e⊤|A||B|e.The denominator avoids cancellation and bounds the checksum functional structurally; numerical cutoffs come from measurement, not γK.Source: §3.1 · pp. 4–5
• Reducing the checksum vector to a scalar made sign and mantissa flips undetectable in 0 of 1,800 samples, while per-row comparison caught 155 of 200 single sign flips on the same workload.The authors retract the earlier claim that the effect was structural and attribute it to the scalar reduction; measured row-residual correlation was 0.000126 on a 2000 × 256 matrix.Source: §3.1 · p. 5

3.2 Perturbation model

• The output-tensor XOR model flips distinct element-bit pairs after GEMM, excluding faults in operands, accumulators, instructions, control flow, and other memory; it does not estimate silicon fault incidence.The authors describe this as a detector characterization under a defined perturbation model, not a characterization of faults in silicon.Source: §3.2 · pp. 5–6

3.3 Workload, hardware, and platform determinism

• All 44 repeated-launch suite cases were bitwise identical on the tested machine, supporting a clean floor attributable to accumulation error rather than kernel-selection variance.This is a measured property of one GPU, driver, CUDA version, and framework build, not a general cuBLAS property.Source: §3.3 · p. 6

3.4 Threshold and detection predicate

• A 10^-6 per-GEMM false-positive target implies an independent-row target of approximately 2.44×10^-10, so the authors extrapolate a Generalized Pareto tail threshold from 20,000 clean GEMMs.The 99th-percentile fit supplies the canonical threshold; detection rates use it unless stated otherwise.Source: §3.4 · pp. 6–7
• The per-GEMM conversion assumes row independence; exceedance counts had a variance ratio of 0.985 against a binomial reference, supporting the conversion at the fitting threshold but not proving tail independence.The canonical threshold falls between the two observed clean maxima, 3.245921×10^-6 and 2.684525×10^-6.Source: §3.4 · p. 7

3.5 Pre-registration

• Predictions were committed with numeric ranges and explicit falsifiers before their corresponding data existed, and failed predictions were reported rather than rewritten.The kill-test bar was unchanged from the repository README written before code existed.Source: §3.5 · pp. 7–8
• Pre-registration did not guarantee measurement correctness: audit caught a faulty detection predicate and a results document citing a nonexistent source-data path.The source file was located, its numbers reproduced, and the provenance corrected in the repository.Source: §3.4 · p. 7, §3.5 · p. 8

4 Results

• Detection rates are scored against τ = 3.144535556×10−6 unless otherwise stated.The detection predicate is rmax > τ or a non-finite rmax.Source: §4 · p. 8

4.1 The clean noise floor scales as K−1/2

• −0.4999 is the fitted exponent for the clean per-row residual across the full 16× K range.The median falls monotonically across 2,500 clean draws with M = N = 4096 fixed.Source: §4.1 · p. 8
• K^-1/2 describes the measured relative-residual scaling, but the original argument conflated absolute and relative error and predicted the wrong direction.The measured exponent matches K scaling in magnitude, but the argument did not supply the reason for that exponent.Source: §4.1 · p. 8

4.2 Detectability does not vary with K

• The flat detection result fired the registered falsifier: the proposed mechanism that detectability degrades with K was wrong.The prediction specified that flat detection across a 16× K range would falsify the mechanism.Source: §4.2 · pp. 8–9
• 79%, 75%, 73%, 81%, and 77% of single sign-bit flips were detected at K = 512 through 8192, with no monotone trend.The mean was 77%, and the eight-point spread was consistent with binomial noise at n = 100.Source: §4.2 · p. 9
• Signal and clean floor scale together, leaving their ratio independent of K; bit position, rather than contraction length, governs detectability.The threshold is derived from the clean floor at each K, which falls as K^-1/2.Source: §4.2 · p. 9
• Per-K observed clean maxima from 500 samples make cross-K comparisons internally consistent, but each maximum is unstable as an absolute threshold.The paper contrasts these sample maxima with τ as the stable absolute reference.Source: §4.2 · p. 9

4.3 Bit position governs detectability

• Exponent flips are caught most often, sign flips slightly less often, high-mantissa flips rarely, and low-mantissa flips never.This ordering is stable across every tested contraction length.Source: §4.3 · pp. 9–10
• The bit-class pattern reflects perturbation magnitude relative to the accumulation floor and does not depend on K.Table 2 gives the full flip matrix at K = 4096, n = 200 per cell, scored at τ and the observed clean maximum for comparison.Source: §4.3 · pp. 9–10

4.4 Low mantissa flips are a hard negative

• No MANTISSA LOW flips were detected: 0/600 at τ with K = 4096, and 0/1500 in the K-scaling study scored at per-K observed clean maxima.The counts use different thresholds and must not be merged.Source: §4.4 · p. 10
• At K = 4096, the residual ratio was identical to four significant figures at one, two, and four low-mantissa flips; more corruption did not produce a larger residual.The ratios were 0.5247, 0.6217, and 0.827 relative to the pilot clean maximum; the paper places the perturbation below the accumulation floor at this K.Source: §4.4 · p. 10
• The low-mantissa flips were not rounded away: all targeted elements changed after casting, and every sample had a positive maximum relative delta.The MANTISSA LOW relative deltas ranged from 0.00407 to 0.0313 across 50 samples per class.Source: §4.4 · p. 10
• The shared minimum ratio of 0.5247 across all six mantissa cells remains unexplained, and a registered prediction that might explain it could not be tested.Source: §4.4 · p. 10

4.5 A calibrated false-positive bound

• The two evaluable Generalized Pareto fits had negative shape parameters with confidence intervals excluding zero, and extrapolated thresholds were 1.17× and 1.24× the observed clean maximum.The fits were reported separately, not averaged.Source: §4.5 · p. 10, Table 4 · p. 11
• The evaluable fits agreed at 1.063×, within the preregistered factor-of-two sensitivity criterion; the third fit could not be evaluated.Source: §4.5 · p. 10
• At the fitting threshold, the per-row exceedance-count variance ratio was 0.985 against a binomial reference, supporting—but not proving—the independence-based conversion in the far tail.The paper carries far-tail independence forward as an assumption.Source: §4.5 · p. 10

4.6 The detector fails its registered bar

• 76% of single exponent-bit flips were detected, below the pre-registered 90% bar at the canonical threshold.The bar required a per-GEMM false-positive rate below 10^-6.Source: §4.6 · p. 11
• At the pilot clean maximum, pooled exponent detection was 301/400 (75.25%); at the K-scaling clean maximum, it was 320/400 (80%).These are sample maxima at different sample sizes; τ is canonical because it alone has a stated false-positive rate and falls between them.Source: §4.6 · p. 11
• The pre-fit analysis predicted that 75.25% bounded detection from above at any defensible threshold, leaving the registered bar unreachable.Detection fell rather than rose after the fit.Source: §4.6 · p. 11

4.7 Registered predictions that failed

• 136/200 = 68% on P5 missed its 70% bar, while Q2 through Q4—the study’s proposed mechanism—failed.The verdict was recorded before the corresponding data existed.Source: §4.7 · p. 11

4.8 Retractions

• The scalar reduction discarded M−1 degrees of freedom; with per-row comparison, sign flips were detected in 155/200 same-workload samples.The earlier scalar evaluation’s 0/1800 sign- and mantissa-flip result was not structural invisibility, and that evaluation is void.Source: §4.8 · p. 12
• The suspected injector artifact behind the MANTISSA LOW zero was not confirmed, so the hard negative stands.The hypothesis was withdrawn after testing.Source: §4.8 · p. 12
• Operand-asymmetry predictions were untestable because injection affected only the output tensor; the shared 0.5247 minimum remains unexplained.Operand-level injection is identified as future work.Source: §4.8 · p. 12
• Correctly counting non-finite residuals makes exponent-flip detection monotone at every tested K, withdrawing the reported non-monotonicity.The prior comparison treated non-finite residuals as undetected; the same defect affected the purported corroborating run.Source: §4.8 · p. 12

5 Scope

• All corruption was injected into the output tensor after GEMM, so the study does not estimate fault incidence in silicon.Operands, accumulator intermediates, instruction encoding, and control flow were not perturbed.Source: §5 · p. 13
• The clean floor comes from one Tesla T4 (sm75), bfloat16 workload, and M = N = 4096 shape family, so it should not transfer to other hardware.The floor is per-configuration by construction.Source: §5 · p. 13
• The p95 fit is not evaluable because top-64 capture truncated all 19,500 GEMMs; the p99 truncation effect was negligible.At p95, roughly two thirds of exceedances were lost; at p99, only 2 of 19,500 GEMMs exceeded capture.Source: §5 · p. 13
• The extrapolation extends from exceedance probability of order 10^-2 to 2.44 × 10^-10, several decades beyond observed data.The fitted upper endpoints were 3.74 × 10^-6 from the p99 fit and 4.60 × 10^-6 from p99.9.Source: §5 · p. 13
• Flip cells contain n = 100 in the K-scaling study and n = 200 at K = 4096, leaving sub-percent detection probabilities unresolved.The corresponding standard errors are roughly four and three percentage points.Source: §5 · p. 13
• The shared 0.5247 minimum ratio across mantissa cells remains unexplained.Source: §5 · p. 13
• Two checksum-reduction backends disagree; all reported results use float64 accumulation, and calibration should not be assumed to transfer to the other backend.The discrepancy remains unresolved.Source: §5 · p. 13

6 Discussion

• At a defensible operating point, the detector achieves 76% on exponent corruption and is blind to low-mantissa corruption; lowering the threshold cannot recover below-floor perturbations because the false-positive requirement bounds it.Source: §6 · p. 13
• The empirical threshold is a quantile tied to a stated false-positive rate, whereas the classical τ = Ku is a bound that is undefined in this regime.The authors describe the quantile as a weaker guarantee but usable where the bound is undefined.Source: §6 · pp. 13–14
• Detectability remains invariant in K because the normalizer grows at the same rate as the noise, contrary to the study’s initial intuition.The authors recommend checking whether normalization cancels predicted sensitivity effects.Source: §6 · p. 14
• Under controlled bfloat16 injection and calibrated thresholds, the findings support checksum ABFT as a cheap first filter with known blind spots, not a complete SDC detector.The paper contrasts its findings with Ma et al. [2025], who studied real corruption on unhealthy nodes at float32.Source: §6 · p. 14

7 Conclusion

• 76% of single exponent-bit flips were caught, below the registered 90% bar, while none of 600 low-mantissa flips were caught.Both results use the calibrated threshold for one-sided checksum ABFT at bfloat16 and K = 4096.Source: §7 · p. 14
• Detectability did not vary with contraction length, falsifying the mechanism the study was designed to demonstrate.The paper concludes that bit position, rather than contraction length, governs detectability.Source: §7 · p. 14
• Four registered predictions failed and four findings were retracted; the threshold, perturbation model, timestamped predictions, and data were released for checking.Two retractions were caught by audit rather than prediction.Source: §7 · p. 14