PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference

Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, Zirui Liu

arXiv:2506.09501v2cs.CL

TL;DR

LLM benchmark reproducibility remains fragile because numerical precision and runtime configuration can change outputs even under supposedly deterministic inference. The paper systematically analyzes these effects across hardware, software, and precision settings, and proposes LayerCast, which computes in FP32 while storing weights in BF16. It finds that FP32 provides near-perfect determinism, whereas BF16 exhibits substantial variance, motivating evaluation practices that explicitly account for numerical nondeterminism.

  • Problem

    The paper addresses limited evidence about how numerical precision and hardware configurations affect reproducibility in LLM inference, a gap that matters because benchmark results can vary across system settings.

  • Method

    The paper conducts experiments across models, tasks, hardware configurations, runtime settings, and precision formats, then develops LayerCast with FP32 computation and BF16 weights.

  • Results

    FP32 provides near-perfect determinism, while widely used BF16 exhibits substantial variance under supposedly deterministic greedy decoding across runtime configurations.

  • Takeaways & Limitations

    Reproducible LLM evaluation should account for numerical precision and runtime nondeterminism, using multiple sampled runs when resources permit or FP32 for single-run greedy decoding.

  • Takeaways & Limitations

    The paper notes that AIME’24 has only 30 problems, so sampling noise can amplify variance and produce an exception to the overall precision trend.

Abstract

from arXiv · show

Large Language Models (LLMs) are now integral across various domains and have demonstrated impressive performance. Progress, however, rests on the premise that benchmark scores are both accurate and reproducible. We demonstrate that the reproducibility of LLM performance is fragile: changing system configuration, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses. This issue is especially pronounced in reasoning models, where minor rounding differences in early tokens can cascade into divergent chains of thought, ultimately affecting accuracy. For instance, under bfloat16 precision with greedy decoding, a reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9% variation in accuracy and 9,000 tokens difference in response length due to differences in GPU count, type, and evaluation batch size. We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision. This work presents the first systematic investigation into how numerical precision affects reproducibility in LLM inference. Through carefully controlled experiments across various hardware, software, and precision settings, we quantify when and how model outputs diverge. Our analysis reveals that floating-point precision - while critical for reproducibility - is often neglected in evaluation practices. Inspired by this, we develop a lightweight inference pipeline, dubbed LayerCast, that stores weights in 16-bit precision but performs all computations in FP32, balancing memory efficiency with numerical stability. Code is available at https://github.com/nanomaoli/llm_reproducibility.

1 Introduction

LLM evaluation can lose reproducibility when numerical precision and runtime configuration vary, even under greedy decoding with a fixed prompt and seed. The paper analyzes this problem and proposes higher-precision computation and an FP32-based pipeline to improve stability.

  • Problem: Numerical precision can undermine deterministic greedy decoding, allowing outputs to differ across hardware and system configurations despite the same prompt and random seed.The paper identifies non-associative floating-point arithmetic and rounding effects as the underlying source of variation.
  • Why reproducibility matters: Single-run benchmarks and standard deviations that ignore numerical nondeterminism can mislead performance comparisons and overestimate true model uncertainty.Reported variance may combine intrinsic model uncertainty with variation introduced by finite numerical precision.
  • Sources of variation: BF16 inference is sensitive to tensor parallel size, evaluation batch size, and GPU type, factors that are often overlooked and may vary with resource availability.The paper highlights these runtime factors as important sources of reproducibility differences.
  • Mitigation: Increasing mantissa bits with FP16 or FP32 can significantly mitigate reproducibility problems caused by limited numerical precision.The paper presents higher precision as a practical way to reduce sensitivity to hardware and system configuration.
  • Mitigation: LayerCast stores model weights in BF16 while performing all computations in FP32, balancing memory efficiency with reproducibility.The pipeline is released as a vLLM patch requiring only a few lines of code changes.

2 Preliminary and Analysis

The paper examines how evaluation practices, floating-point precision, rounding, and GPU execution order affect LLM reproducibility. It explains how non-associative arithmetic can alter logits and token choices, while runtime kernels and batching change operation order.

  • Evaluation practices: Greedy decoding selects the highest-probability token at each step, whereas random sampling uses non-zero temperature and commonly evaluates mean accuracy across independent trials.These are the two widely adopted evaluation strategies described in the paper.
  • Evaluation practices: Fixed random seeds and deterministic-library settings do not guarantee full reproducibility because computation order can still vary across runs.The paper notes that framework flags can avoid some nondeterministic algorithms but do not eliminate all nondeterministic results in practice.
  • Numerical precision: Lower-precision formats introduce larger rounding errors than FP32 when representing the same value.The paper uses floating-point formats and illustrative tables to connect precision with rounding error.
  • Numerical precision: Floating-point addition is non-associative, so changing summation order can produce different numerical results, especially under BF16.Higher-precision FP32 is more tolerant of rounding errors in the illustrated cases.
  • GPU execution: Small logit differences can change token selection because GPU reductions execute attention and logit computations in configuration-dependent orders.Thread scheduling and parallel execution can alter the order of floating-point operations.
  • GPU execution: Continuous batching, operator implementations, kernel hyperparameters, collective operations, and tensor parallelism can all change floating-point execution order.These mechanisms provide concrete routes by which serving-system configuration affects inference determinism.

3 Reproducibility Issues with Limited Numerical Precision

Across controlled evaluations, limited numerical precision makes both greedy and random-sampling outputs sensitive to hardware, batch size, GPU count, and GPU architecture. FP32 substantially improves reproducibility, while BF16 can produce large accuracy, length, and divergence variability, especially in reasoning models.

  • Experiment setup: Experiments cover four models, five benchmarks, and 12 runtime configurations combining two GPU types, two GPU counts, and three batch sizes.The study evaluates both greedy decoding and random sampling to quantify output instability across deployment-like settings.
  • Metrics: The evaluation measures accuracy variance, output-length variance, divergence position, top-1 probability variance, and sampling-based Pass@1 stability.Div_Index measures where responses first diverge, while Avg_Std@top1_prob tracks prediction-probability variation before divergence.
  • Greedy decoding: 9% standard deviation in AIME’24 accuracy occurs for DeepSeek-R1-Distill-Qwen-7B under BF16, versus virtually zero variance under FP32.The same experiments show that changing system configuration can also alter reasoning-model response lengths by up to 9,000 tokens.
  • Greedy decoding: Small probability gaps between competing tokens allow numerical errors to flip top-1 selection, causing divergent greedy-decoding sequences.The paper attributes the instability to BF16’s limited mantissa, which increases probability-computation errors and token flips.
  • Precision effects: FP32 produces fewer divergent examples and later divergence points than BF16, with near-zero top-1 probability variance across evaluated settings.FP16 is intermediate: it shows lower variance than BF16 but greater variability than FP32.
  • Random sampling: Random-sampling evaluations also vary with numerical precision, and BF16 may require substantially more runs to achieve comparable statistical confidence.An AIME’24 exception at n = 64 shows FP32 variance of 0.7377 versus BF16 variance of 0.3749, attributed to the small dataset and sampling noise.
  • Runtime configurations: GPU count, batch size, and GPU architecture alter token-probability variation, with effects strongest under BF16 precision.Four GPUs and A100s generally show higher variation, while smaller batches can also increase variance in the tested settings.
  • Conclusion: Numerical precision, rather than greedy decoding alone, is therefore critical for reproducible LLM inference across runtime configurations.The paper identifies small competing-logit gaps, precision format, and runtime configuration as interconnected sources of output variability.

4 Near-Perfect Deterministic Reproduction: LayerCast

LayerCast addresses the cost of full-FP32 inference by storing weights in 16-bit precision while computing in FP32. Experiments indicate stability comparable to FP32, reduced memory use, and low divergence across configurations.

  • LayerCast design: LayerCast stores weights in BF16 and upcasts each weight to FP32 just before matrix multiplication.This hybrid approach is designed to balance computational stability with memory efficiency.
  • LayerCast design: LayerCast performs all inference computations in full FP32 precision while retaining memory-efficient 16-bit weight storage.The design preserves FP32 computational stability while keeping the memory footprint closer to that of 16-bit models.
  • Experimental results: LayerCast achieves accuracy stability nearly identical to FP32, whereas BF16 exhibits substantially higher variability across runs.The comparison uses the standard deviation of accuracy across runs.
  • Experimental results: Divergence rates remain below 3.4% across different batch sizes and GPU configurations, while memory usage is reduced by 34% versus full FP32.The memory reduction is particularly relevant to KV-cache requirements in long-context scenarios.

5 Related Works

Prior work identifies reproducibility as a persistent challenge in LLM inference and studies several sources of output instability. Figure 8 compares Div_Index distributions for DeepSeek-R1-Distill-Qwen-7B on MATH500 under BF16, FP32, and LayerCast.

  • Prior reproducibility studies: Figure 8 compares Div_Index distributions for DeepSeek-R1-Distill-Qwen-7B on MATH500 under BF16, FP32, and LayerCast.
  • Prior reproducibility studies: Existing studies report that nondeterministic behavior during LLM inference is widely observed.
  • Prior reproducibility studies: Prior research examines prompt formatting, decoding parameters, random seeds, and hardware and software configurations as factors affecting reproducibility.

6 Conclusion

The paper concludes that numerical precision is a major reproducibility issue in LLM inference and that LayerCast offers FP32-level determinism with lower memory overhead. It also calls for evaluation practices that account for precision effects.

  • Conclusion: Experiments across models, tasks, and hardware configurations show that greedy-decoding outputs can vary because floating-point arithmetic is non-associative.
  • Conclusion: FP32 provides near-perfect determinism, while widely used BF16 exhibits substantial variance.
  • Conclusion: LayerCast achieves FP32-level determinism while maintaining reasonable memory efficiency.
  • Conclusion: The findings highlight the importance of standardizing evaluation practices to account for numerical precision effects.

A.1 MATH Benchmarks: AIME’24 and MATH500

The benchmark materials include prompts for mathematical problem solving with model-specific templates. The supplied templates distinguish response formats and testing modes, including boxed answers and stdin or functional execution.

  • Models and prompts: The benchmark materials reference DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B.
  • Models and prompts: One prompt template inserts a problem text placeholder and requests the final response within a boxed expression.
  • Testing modes: The coding-oriented templates distinguish stdin tests, which require reading input and printing output, from functional tests, which return a function body without invoking it.

B Supplementary Results on Accuracy Variance under BFloat16

Under BF16 greedy decoding, accuracy reproducibility varies with dataset difficulty and runtime configuration. Across the evaluated metrics, increasing precision from BF16 to FP32 reduces divergence and instability.

  • Accuracy variance: Accuracy variance under BF16 is modest on MATH500 and LiveCodeBench-Easy but substantially larger on LiveCodeBench-Medium and LiveCodeBench-Hard.The results indicate that numerical-precision effects on reproducibility scale with problem complexity.
  • Evaluation metrics: The evaluation measures Std@Acc, Avg_Std@Output_Length, Div_Index, and Div_Percent across 12 runtime configurations.These metrics capture accuracy variation, output-length variation, token-level agreement, and divergent-output frequency.
  • Accuracy variance: Figure 9 summarizes accuracy variation across different settings under BFloat16.The figure extends the configuration-variance analysis across the evaluated datasets.
  • Precision effects: As precision increases from BF16 to FP32, Std@Acc and Div_Percent decrease while Div_Index increases.The reported pattern consistently indicates reduced divergence under higher numerical precision.

C.1 Additional Results on Larger Models and Diverse Tasks

Additional experiments show that numerical-precision effects on reproducibility extend to larger models, graduate-level science tasks, and a non-vLLM inference backend. Across these settings, FP32 is more consistent than BF16.

  • Larger models: Qwen3-32B on AIME’24 shows Std@Acc of 5.02% under BF16 versus 1.11% under FP32 across runtime configurations.The configurations vary GPU count, GPU version, and batch size; only 9 of 12 FP32 configurations are feasible because of memory constraints.
  • Diverse tasks: DeepSeek-R1-Distill-Llama-8B on GPQA Diamond shows Std@Acc of 3.03% under BF16 versus 1.96% under FP32.GPQA Diamond evaluates graduate-level science questions, extending the analysis beyond math and coding tasks.
  • Cross-setting pattern: Across the supplementary evaluations, BF16 consistently exhibits higher variance than FP32 across model scales and task domains.The larger-model and diverse-task experiments support the same reproducibility pattern observed in the main evaluations.
  • Backend validation: With HuggingFace Transformers, FP32 produces 0% Std@Acc for both tested models, while BF16 produces 2.23% and 1.51%.The experiments use 12 settings and reproduce the precision-dependent pattern without vLLM or PagedAttention.
  • Backend validation: The backend comparison indicates that the reproducibility phenomenon is observable across different inference backends.The reported results attribute the phenomenon to GPU hardware and floating-point arithmetic rather than to vLLM alone.

D Supplementary Results on Ablation Study

Ablation experiments isolate runtime factors associated with numerical instability during greedy decoding. Their effects differ by GPU type and metric, while larger batch sizes generally improve reproducibility.

  • GPU count: Increasing L40S GPUs from 2 to 4 generally increases Avg_Std@top1_prob, indicating greater inference nondeterminism.The corresponding trend is less consistent on A100 GPUs, where 2-GPU results are often slightly higher than 4-GPU results.
  • GPU count: The GPU-count analyses separate L40S and A100 because the effect of GPU count depends on GPU version.This separation is necessary because A100 shows higher instability in the reported experiments.
  • Batch size: Larger batch sizes consistently produce lower Avg_Std@top1_prob across the tested GPU counts and GPU types.The authors relate this pattern to reduced error accumulation with higher parallelism.
  • GPU version: A100 GPUs consistently show slightly higher Avg_Std@top1_prob than L40S GPUs under the same settings.In these experiments, the authors associate the higher A100 variability with less stable top-1 token predictions.
  • Metric interpretation: Avg_Std@top1_prob is more informative than Std@Acc for these ablations because it directly reflects fluctuations in predicted token probabilities.Probability fluctuations do not always cause token flips, and token flips do not always change final accuracy.

E Supplementary Results on Random Sampling

Under random sampling, FP32 generally yields the most stable results across runtime configurations, but FP16 is more stable in some cases. The findings therefore support using more extensive sampling for reproducibility.

  • Precision comparison: FP32 yields the lowest standard deviations in most random-sampling cases across the evaluated models and datasets.The results summarize Pass@1 accuracy and standard deviation across precision, batch-size, and GPU-count settings.
  • Precision comparison: FP16 is more stable than the other precisions in 4 out of 12 cases.The authors attribute this exception pattern to interleaving between two sources of randomness during random sampling.
  • Practical implication: The authors recommend more extensive sampling to obtain more reproducible results under random sampling.This recommendation follows the observed variation across precision settings and runtime configurations.

F Supplementary Results on LayerCast

Supplementary evaluations test LayerCast across models, benchmarks, hardware, GPU counts, batch sizes, and numerical precisions. Across these settings, LayerCast matches FP32 stability while retaining memory-efficiency benefits, whereas BF16 remains variable.

  • Evaluation settings: LayerCast evaluations span models and benchmarks across batch sizes 8, 16, and 32, GPU counts 2 and 4, GPU types A100 and L40S, and BF16, FP32, and LayerCast precision settings.The experiments use the same configuration as the main studies.
  • Method: LayerCast performs all computations in FP32 while retaining BF16 weights, addressing the rounding-error accumulation identified as the source of nondeterminism.Just-in-time casting from BF16 storage to FP32 computation is deterministic and introduces no additional variance.
  • Qwen-7B results: Across five benchmarks, LayerCast matches FP32 stability for DeepSeek-R1-Distill-Qwen-7B, often producing zero or near-zero standard deviation while preserving memory efficiency.BF16 exhibits significantly higher standard deviations in the same comparison.
  • Llama-8B results: 0% Std@Acc is achieved by LayerCast across all 12 runtime configurations for DeepSeek-R1-Distill-Llama-8B on AIME’24, matching FP32.The result maintains memory-efficiency benefits across the tested model architecture and configurations.
  • Llama-8B results: BF16 shows Std@Acc of 5.87% on A100 configurations and 3.44% on L40S configurations for DeepSeek-R1-Distill-Llama-8B on AIME’24.These results contrast with LayerCast’s perfect reproducibility across the same 12 runtime configurations.
Loading 2506.09501v2…