Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance

Teng-Ruei Chen

Source-linked AI summary

Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance

Teng-Ruei Chen

arXiv:2609.00363v1Aug 25, 2026cs.LGcs.SE

TL;DR

Tolerance-based conformance can verify arithmetic preconditions, operand sharing, and bounded differences without certifying interchangeability when epilogue faults remain within one bfloat16 spacing. This paper measures that boundary with fault injection and shows that requantized power-of-two scales enable byte-identical cross-kernel inference at measured perplexity costs.

Problem
The paper asks whether tolerance-based conformance tests can detect faulty quantized GEMM implementations rather than merely agree with presumed-good kernels.
Method
The study injects nine known fault families into an exact W8A8 reference pipeline and evaluates seven checks across 8,232 layer–fault–regime cells, then tests requantized power-of-two scales.
Results
Requantized power-of-two scales produce bitwise per-layer agreement and byte-identical generations across three model sizes, with observed perplexity point estimates from −0.28% to +0.48%.
Takeaways & Limitations
The suite establishes preconditions, operand provenance, and one-spacing boundedness, while power-of-two scales make the equality-based check deployable.
Takeaways & Limitations
Evidence covers one model family, one CUTLASS–Triton kernel pair, Qwen3 at three sizes, and selected serving conditions, with some measurements limited to 1.7B or unmeasured.

Abstract

from arXiv · show

Conformance suites for quantized GEMM kernels ask whether two implementations agree within a tolerance. We measure what such a suite can detect. Injecting nine faults into a reference INT8 pipeline over 8,232 layer--fault--regime cells of Qwen3-1.7B, we find that every one of five epilogue faults -- scale precision, double rounding, multiplication order, output truncation, fused ordering -- moves the output by at most a single bfloat16 spacing, and by exactly one whenever it moves it at all, across 5,880 cells. A tolerance of one spacing is therefore blind to the entire class by construction: four of the five faults are detected by no check in the suite, and the fifth only under power-of-two scales. Faults that violate the accumulator's exactness preconditions, or that break operand sharing, are detected without exception, and a null fault never fires. What a tolerance-based suite of this shape establishes is therefore narrower than interchangeability: that the preconditions hold, that operands are shared, and that differences stay within one spacing. The power-of-two constraint that exposes the one detected fault is also deployable. Requantizing every weight scale to its nearest power of two makes CUTLASS and Triton agree bitwise at every linear layer (196/196 and 252/252, against 8/196 and 10/252 under the checkpoints' own scales) and yields byte-identical generated token sequences at 1.7B, 8B and 14B (8/8 prompts, against 0/8 at all three). Observed perplexity point estimates are +0.32%, -0.28% and +0.48%; the 90% intervals cover zero at the two smaller sizes but not at 14B, reaching +0.71% and +0.76%. A previously reported +157% perplexity for this intervention was an artifact of a probe that rewrote scales without requantizing the weights; separating the effects attributes 99.8% of it to the resulting weight--scale mismatch rather than to the power-of-two constraint itself.

I. INTRODUCTION

The paper tests what a tolerance-based conformance suite can detect and evaluates whether requantized power-of-two scales make cross-kernel determinism deployable. It finds that the suite is structurally blind to several epilogue faults, while the scale constraint enables bitwise agreement at modest observed perplexity costs.

Motivation: The study addresses both the unmeasured sensitivity of the earlier suite and the earlier probe’s lack of a servable checkpoint.The earlier study had tested only presumed-good kernels and reported +157% perplexity for its probe checkpoint.
Contributions: Four of five epilogue faults are detected by no check because each remains within one bfloat16 spacing, the suite’s smallest legal tolerance.The suite therefore cannot certify bitwise epilogue equivalence from this tolerance.
Contributions: 8,232 layer–fault–regime cells were evaluated using nine injected fault families and a pre-registered, amended prediction matrix.The corrected matrix produced zero false positives and zero false negatives.
Contributions: The suite reliably tests arithmetic preconditions and operand provenance, but its tolerance-based epilogue checks do not establish kernel interchangeability.Its supported conclusion is narrower: preconditions hold, operands are shared, and differences remain within one spacing.

II. BACKGROUND AND RELATED WORK

The paper grounds its analysis in an exact W8A8 accumulation pipeline and the power-of-two commutation property. It also defines comparison dimensions for bitwise agreement, ground-truth testing, deployable cost, and pre-registered measurement.

Pipeline and alibi: A W8A8 linear layer multiplies int8 activations and weights, accumulates in INT32, applies activation and weight scales, and rounds the result to bfloat16.The pipeline is expressed as Y = (AW^⊤) · s_as_w.
Pipeline and alibi: Under the verified bound, any tiling, split-K, or reduction tree produces the identical INT32 accumulator, so cross-kernel differences arise only in the epilogue.This isolates scale application and final rounding as the relevant sources of disagreement.
Pipeline and alibi: For finite normal values, power-of-two scales make the two legal multiplication orderings bit-identical before the final bfloat16 cast.The commutation property turns bitwise equality into a required outcome under the stated conditions.
Comparison framework: Table I distinguishes bitwise comparison, ground-truth fault evaluation, deployable cost, and pre-registration as separate dimensions of the paper’s positioning.These labels connect output granularity, known fault presence, accuracy cost, and pre-measurement protocol control.

B. Related Work by Treatment Axis

The related work is organized around inference nondeterminism and testing or formal verification of numerical implementations. The paper’s focus is the narrower question of fault sensitivity and deployable cross-kernel agreement.

Inference reproducibility: Prior inference work documents numerical deviations, nondeterminism, batch dependence, tensor-parallel variation, and verification of served outputs.These studies motivate reproducibility concerns across deployed inference systems.
Numerical testing: Differential testing, operator precision testing, equivalence checking, SMT hardware models, and direct tensor-core probing expose or formalize disagreements between implementations.The paper differs by evaluating a conformance suite against injected faults with known ground truth.

E. Constrained Quantization Scales

The paper situates power-of-two scales among prior constrained-quantization methods while measuring their accuracy cost for cross-kernel agreement. Its measurement instrument combines an exact reference pipeline, injectable faults, checks, and a pinned prediction matrix.

Constrained scales: Prior work uses dyadic scales, power-of-two scale fitting, accumulator-aware quantization, and admissible reduction-depth analysis for integer or low-bitwidth inference.This paper applies the constraint to cross-kernel bitwise agreement and measures its accuracy cost on served checkpoints.
Measurement design: Append-only pre-registration fixed the fault catalogue, checks, thresholds, and prediction matrix before full measurement, with disclosed corrections re-pinned beforehand.This provides an audit trail for the three post-data corrections.
Measurement design: The measurement instrument combines an exact-by-construction reference pipeline, nine injectable faults, seven checks, and a prediction matrix.The reference accumulator uses float64 or INT64 paths whose agreement was verified on the real layer shapes.

B. Nine Faults

The study evaluates nine injected fault families across multiple severities and scale regimes against seven conformance checks. Its preregistered matrix distinguishes expected fires, silences, non-applicability, and predicted misses while separating applicability from verdicts.

Fault catalogue: 8,232 layer–fault–regime cells cover nine faults at three severities and under checkpoint-native or power-of-two scales.The catalogue uses one output element, one percent, or all elements as coverage severities across 196 layers.
Checks: Seven checks test shared operands, accumulator exactness preconditions, power-of-two identity, real-scale tolerance, and token-level risk.The suite includes falsifiable checks and two tolerance-based checks, with one-spacing tolerance as the smallest nonzero allowance.
Scoring rule: Applicability and verdicts are recorded separately, and cells with no changed output are excluded from detection denominators.This prevents non-applicable checks or unobservable injections from being counted as successful silence.
Prediction matrix: 63 pre-data check–fault predictions expanded to 77 cells after a disclosed smoke run corrected three predictions before the full execution.The original matrix recorded eight expected fires, 46 silences, nine non-applicable cells, and five predicted misses.
Prediction matrix: Every predicted fire concerned a precondition or operand fault, while no epilogue fault was predicted to trigger any check.The matrix therefore committed in advance to testing whether the suite could miss locally plausible epilogue errors.

E. Three Cells Corrected After Seeing Data

Three prediction cells were corrected after a disclosed smoke run contradicted their original reasoning, and all corrections moved in the suite’s favour. The examples expose distinct mistakes about rounding, relative error, and operand provenance.

Matrix correction: Three post-smoke corrections flipped 1,278 of 8,232 cells from error to agreement, while the original matrix implied 75.0% detection and substantial scale-check false positives.The corrected matrix was re-pinned before full execution, separating preregistered expectations from post-data changes.
F4 × power-of-two identity: F4 under power-of-two identity changed from silent to fire because exactness in float32 does not prevent bfloat16 rounding differences.Power-of-two scales eliminate intermediate-precision and rounding-order faults F1 and F2, but not a wrong rounding mode; check 5 detects that case.
F9 × real-scale tolerance: F9 under real-scale tolerance changed from silent to fire because one int8 activation shift produced measured distances of 366 to 384 ULP near cancellation.The original quarter-spacing argument considered absolute change but missed large relative changes in near-zero entries.
F9 × exact accumulator: F9 under exact-accumulator checking changed from fire to silent because the injected difference altered a scale while leaving the int8 tensors and accumulator unchanged.The other two severity levels retained different accumulators, requiring the original row to be split.

IV. HOW MUCH CAN A TOLERANCE SEE?

The corrected scorecard reports perfect agreement with the preregistered predictions, but the detection table shows that most epilogue faults remain invisible. The suite reliably catches precondition violations and operand faults instead.

Scorecard: Zero false negatives and zero false positives occur across 8,232 cells against the corrected matrix, while null fault F8 never fires in 392 cells.The clean scorecard alone does not reveal which injected faults were observable yet undetected.
Epilogue faults: 4,704 cells containing four of five epilogue faults are detected by no check across either scale regime or any severity.This includes injections corrupting every output element.
Epilogue faults: 494 truncation cells are detected only under power-of-two scales, whereas 478 observable truncation cells under checkpoint-native scales trigger no check.The power-of-two identity check is the sole detector for the truncation fault.
Precondition faults: Both precondition violations are caught in every cell, contrasting with the broad blindness to epilogue faults.The detection asymmetry motivates interpreting the suite as a test of exactness assumptions and operand sharing rather than general interchangeability.

B. Why: One Spacing Is the Whole Budget

The tolerance check cannot distinguish epilogue faults that move outputs by at most one bfloat16 spacing, because one spacing is already the smallest legal tolerance. The suite therefore supports boundedness and precondition claims, not interchangeability.

Spacing budget: 5,880 epilogue cells have maximum ULP distances only 0 or 1, and every observable epilogue fault moves exactly one spacing.The operand fault instead reaches distances from 68 to 35,571, separating the two fault classes structurally.
Prediction matrix: Table III encodes predicted fires, silences, non-applicability, predicted misses, and post-data corrections across seven checks.Its check columns cover shared operands, exactness preconditions, power-of-two identity, real-scale tolerance, and token-level risk.
Detection accounting: Table IV separates observable cells from detected cells, excluding injections that changed no output element from the detection denominator.This distinction prevents unobservable faults from being credited as correct silence.
Spacing budget: The one-spacing tolerance cannot be tightened without becoming the power-of-two identity check, which requires a different checkpoint.Under power-of-two scales, the identity check exposes the truncation fault rather than merely tightening a real-scale threshold.
What survives: The suite establishes exactness preconditions, shared operands, and one-spacing boundedness, but a kernel with F1, F2, F3, or F5 passes every check.The token-level check remains unmeasured because layer-level injections provide no logit margins.

V. POWER-OF-TWO SCALES AS A DEPLOYABLE CONDITION

Requantizing weights under a nearest-power-of-two scale makes the CUTLASS and Triton kernels agree bitwise, while keeping the measured accuracy cost small. This condition also turns the epilogue comparison into a required equality under the stated arithmetic preconditions.

Checkability: Power-of-two scales make the epilogue comparison a required equality instead of a one-spacing tolerance under the stated arithmetic preconditions.This is the condition under which the suite can test the epilogue more strictly.
Requantization: The correct construction requantizes int8 weights under the constrained scales rather than rewriting scales while leaving weights unchanged.The unconstrained reconstruction had to reproduce the original checkpoint byte for byte before constrained arms were built.
Cross-kernel determinism: 8/8 pinned prompts produce byte-identical token sequences at 1.7B, 8B, and 14B, versus 0/8 under the checkpoints’ own scales.The sequences use greedy generation for 64 tokens per prompt.

C. The Cost, and the Decomposition of +157%

The nearest-power-of-two constraint has a small observed perplexity cost, while the previously reported +157% degradation came primarily from an invalid probe construction. The evidence supports cross-kernel determinism but remains bounded by the evaluated models, kernels, and measurements.

Decomposition: 99.8% of the reported +157% degradation is attributed to the weight–scale mismatch created by rewriting scales without requantizing weights.The correctly requantized nearest arm costs +0.32%.
Decomposition: +0.54% is the ceiling arm’s perplexity cost, exceeding the nearest arm’s +0.32% cost.The ceiling arm never shrinks a scale, while nearest permits some outlier clipping in exchange for finer resolution.
Implication: The constraint yields cross-kernel bitwise determinism and replaces the tested one-spacing epilogue tolerance with a required equality under the stated preconditions.This is the paper’s deployability and checkability trade-off.
Scope: The evidence covers Qwen3 at three sizes and the CUTLASS–Triton pair, while throughput was measured only at 1.7B and no decode-regime per-layer capture exists.Above 14B, tensor-parallel reductions confound the single-kernel treatment; 14B evidence is end to end only.

VII. CONCLUSION AND FUTURE WORK

A one-spacing tolerance cannot certify bitwise epilogue equivalence, so the suite establishes only arithmetic preconditions, operand provenance, and bounded differences. Requantized power-of-two scales provide a deployable equality mechanism, while future work targets broader kernels, regimes, and fault families.

A one-spacing tolerance misses four epilogue fault families by construction and cannot certify bitwise equivalence.The tolerance is the smallest that admits any legal difference, so tightening it does not solve the underlying issue.
The suite’s supported claim is limited to preconditions, operand provenance, and one-spacing boundedness rather than interchangeability.
Power-of-two scales make the equality check deployable by removing scale-application freedom under the stated arithmetic preconditions.
Future work includes a second kernel pair, 14B and decode-regime captures, end-to-end token-check injection, and faults exceeding one spacing.
The artifact record includes amendments, prediction matrices, injectors, regression tests, requantization code, measurement artifacts, and run logs, while the repository remains private until announcement.