PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism?

Alizishaan Khatri

arXiv:2609.31796v1cs.LGcs.AI

TL;DR

Activation probes may be trained under one inference configuration and served under another, although batch size and numerical precision can change the activations they receive. This paper evaluates probe transfers and per-example verdict agreement across configurations, finding that prompt verdicts are stable while decoding-time changes largely track whether generated tokens diverge. It concludes that monitor evaluations should measure per-example agreement and distinguish representational noise from input change.

  • Problem

    It is not established whether inference-time batch-size and precision changes, which alter activations, also disrupt probes deployed on those activations.

    Source: §1 · p. 2

  • Method

    The study measures activations and evaluates probes trained on one configuration across other batch-size and precision configurations, comparing verdicts example by example.

    Source: §3 · pp. 4–5

  • Results

    At the prompt, probe accuracy moves by at most 0.47 pp and only 0.076% of verdicts change, while decoding-time verdict changes occur mainly on rows whose generated tokens diverged.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Activation-monitor evaluations should report per-example agreement and separate representational noise from input-text change.

    Source: §1 · p. 2, §7 · p. 9

    The study did not measure how much a probe would move under a different training seed.

    Source: §3 · p. 5

Abstract

from arXiv · show

Activation probes are increasingly used to monitor LLMs in deployment. A probe is typically trained under one inference configuration, then used under whatever batch size and numerical precision the serving stack uses. Because common GPU kernels are not batch-invariant and floating-point formats round differently, the activations seen at deployment are not the ones the probe was trained on. We measure what that costs for Llama-3.1-8B, Qwen3-8B and Gemma-3-4B across batch sizes 4, 8 and 16 and float32, bfloat16 and float16, training 768 probes on one configuration, evaluating each on every other, and comparing verdicts example by example. Probes are stable, but aggregate accuracy is the wrong instrument for showing it: it understates how many verdicts change by a factor of two to nine. At the prompt, accuracy never moves by more than 0.47 percentage points across 1,392 transfers and only 0.076% of verdicts change; under float32 with only the batch size varied, none of 201,960 verdicts change. During decoding the flip rate rises to 2.8%, but rows whose realised tokens matched flip in only 0.12-0.15% of cases, while rows whose tokens diverged flip in 12.9%: the cause is the text, not the arithmetic. A bfloat16 batch-size change flips the first generated token for 2.1% of rows and leaves 25% on different tokens by token 20. Flips are symmetric, Cohen's kappa stays above 0.94, and AUROC moves by at most 0.05 points. Underneath, activations move about as much as the format's rounding: a bfloat16 batch-size change perturbs them by a median relative L2 of 1e-2, roughly 8x the float16 figure. Probes absorb this; the model's own next-token argmax does not. Robustness evaluations of activation monitors should report per-example agreement rather than aggregate accuracy, separate representational noise from input change, and state the serving configuration.

1 INTRODUCTION

Activation probes are moving into deployment, where serving batch size and numerical precision can change the activations from those used in training. This study tests whether those changes alter probe verdicts, finding that fixed-input probes remain stable while generated-text changes are associated with most decoding-time flips.

1 INTRODUCTION

  • A probe trained under one model, task, and precision may encounter different activations at serving because batch size and numerical precision change inference-time numbers.The cited background attributes batch-size effects to batch-shape-dependent kernel reductions and precision effects to different floating-point rounding.Source: §1 · p. 2
  • The study compares activations and 768 probes across three models, three datasets, three batch sizes, three precisions, four depths, and four token positions.The design includes the last prompt token and three greedy-decoding steps.Source: §1 · p. 2
  • 0.076% of prompt verdicts change across transfers, while accuracy moves by at most 0.47 percentage points; under fp32, no verdict changes in 201,960 evaluations.During decoding, verdicts change mainly on rows whose generated tokens diverged, not rows whose tokens matched.Source: §1 · p. 2
  • For a fixed input, prompt probes transfer across batch sizes and precisions; during autoregressive generation, configuration-dependent token choices change the probe’s input text.The paper distinguishes benign representation changes, behaviorally consequential discrete choices, and validity failures where a format cannot represent activations.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

Prior work studies numerical non-determinism in LLM outputs and weights, precision-related activation issues, and probe behavior across representations. This paper focuses on inference-time numerical shifts in the intermediate representations consumed by probes.

2 BACKGROUND AND RELATED WORK

  • Prior studies connect batch-shaped GPU reductions and lower precision with changed greedy outputs, while other work documents run-to-run evaluation and training variation.The cited literature also reports that implementation-level training non-determinism can produce variance comparable to changing the random seed.Source: §2 · pp. 2–3
  • fp16 has 5 exponent and 10 mantissa bits with a maximum of 65,504, while bf16 has 8 exponent and 7 mantissa bits and fp32’s dynamic range.Prior work describes large residual-stream activations that can exceed fp16’s range; this study instead varies compute format and batch shape while keeping weights fixed.Source: §2 · p. 3
  • Prior probe research examines selectivity, representational interpretation, and generalization; Khatri & Chan (2026) report prompt activations largely unperturbed by batch size.The authors say their work corroborates and extends that prompt finding to decoding, and that prior work had not isolated inference-time numerical shifts’ effects on probe behavior.Source: §2 · p. 3

3 EXPERIMENTAL SETUP

The experiments compare model activations and probe transfers across a controlled configuration grid, with explicit exclusions and measures for numerical and token-level divergence. Hardware differences constrain some precision comparisons, and probe sensitivity to a different training seed was not measured.

3 EXPERIMENTAL SETUP

  • The study uses Llama-3.1-8B, Qwen3-8B, and Gemma-3-4B with Rotten Tomatoes, SMS Spam, and TruthfulQA; TruthfulQA is used only for activation analysis.Inputs are raw text without a chat template, left-padded to 128 tokens; SMS Spam and TruthfulQA use content-hash splits.Source: §3 · p. 3
  • Runs cover batch sizes 4, 8, and 16 and fp32, bf16, and fp16, using a fixed seed and deterministic algorithms; deterministic mode does not make kernels batch-invariant.Most runs use an L4; Llama and Qwen fp32 use an A100, so their fp32-versus-lower-precision comparisons change both GPU and precision.Source: §3 · p. 3
  • Gemma-3-4B fp16 is excluded because hidden states and logits become NaN, leaving results for Gemma at fp32 and bf16 only.The affected NaNs occur at blocks 17, 26, and 34 across positions, datasets, and batch sizes; the exclusion also removes comparisons involving that configuration.Source: §3 · p. 4
  • Activation comparisons use row-level distances and similarities plus population-level geometry measures; decode-position activation divergence is defined by block-3 relative ℓ2 above 0.1.The criterion detects rows processing different tokens at that position, not the step at which their trajectories first diverged.Source: §3 · p. 4
  • Token divergence is measured by comparing greedy argmax tokens recovered from stored logits, with first disagreement located only among logged positions 5, 10, and 20.A disagreement that appears and self-corrects between logged positions is not observed; the probe sweep also compares verdicts across runs for the same rows.Source: §3 · p. 4
  • The study trains 768 probes across three families and evaluates each on available precision and batch-size cells; probe-training differences across seeds were not measured.The rerun on stored activations reproduced accuracy and macro-F1 exactly, while matched accuracies differed by at most 0.4 pp between training precisions.Source: §3 · pp. 4–5
  • Figure 1a shows median prompt-token activation distance pooled over layers and datasets, while Figure 1b shows the percentage of rows with different realised tokens by position.The caption notes that Llama and Qwen fp32 runs are bitwise identical across batch sizes and that Gemma has no fp16 comparisons.Source: Figure 1 · p. 5

4 HOW MUCH DO ACTIVATIONS CHANGE?

Prompt activations change at rounding scale, with noise that does not drift or amplify substantially through depth; decoding can turn rounding differences into different generated text. The resulting token divergence substantially alters internal-state similarity, while prompt-level global geometry remains largely preserved.

4.1 PREFILL: SMALL, ROUNDING-SCALE, STRUCTURELESS NOISE

  • Under fp32, Llama and Qwen prefill activations are 99.9–100% bitwise identical across batch sizes on the A100; Gemma differences are at fp32-rounding scale.For Gemma, median relative ℓ2 is approximately 5 × 10−7; the GPU is fixed in both comparisons, but model and GPU vary together across them.Source: §4.1 · p. 5
  • At prefill, bf16 batch-size changes perturb Gemma activations by median relative ℓ2 4.9 × 10−3, slightly above the 4.4 × 10−3 fp32-to-bf16 change.This comparison holds the GPU fixed; the 8B comparisons also point this way but change GPU and precision together.Source: §4.1 · p. 5
  • A batch-size change produces median relative ℓ2 of 1.7 × 10−3 in fp16 versus 1.3 × 10−2 in bf16 for Llama and Qwen, about 8× less noise.Both sides of these comparisons use the L4; both formats differ by 2–4 representable steps per element.Source: §4.1 · p. 5
  • Across non-fp32 batch-size triples, prefill noise fits independent rounding draws in 224/224 cases; median per-block growth is 1.01, with no exponential amplification.Relative ℓ2 is nearly unrelated to input norm or sequence length (median R2 = 0.05).Source: §4.1 · p. 6

4.2 DECODE: ROUNDING NOISE BECOMES DIFFERENT TEXT

  • A bf16 batch-size change flips the first generated token for 2.1% of rows and leaves 25% on different tokens by token 20; fp16 rates are 0.4% and 3.6%.Under fp32, changing batch size leaves every realised token identical across all four positions in all 18 pairs.Source: §4.2 · p. 6
  • Among diverging rows, 8.3% already differ at the first generated token and 33.0% by token 5.The block-3 activation-distance threshold agrees with token-id checks to a mean absolute difference of 0.95 pp over 26 cells (r = 0.993).Source: §4.2 · p. 6
  • By token 20, median CKA falls to 0.64 and kNN overlap to 0.45 under bf16 batch-size changes, while prefill CKA is at least 0.995.At prefill, 10-NN overlap is 0.95 under bf16 and 0.99 under fp16, with a minimum of 0.70.Source: §4.2 · p. 6, §4.1 · p. 6

5 HOW MUCH DO PROBES CHANGE ACROSS CONFIGURATIONS?

Probe accuracy changes little at prompt positions, but it can conceal many individual verdict flips. During decoding, flips rise with token divergence, while remaining largely symmetric and leaving ranking metrics comparatively steady.

5 HOW MUCH DO PROBES CHANGE ACROSS CONFIGURATIONS?

  • At decode tokens 5, 10 and 20, accuracy changes in 69%, 75% and 80% of cells, while mean |∆| rises from 0.19 to 0.37 pp.The maximum change is 2.35 pp; changes are nearly balanced, with mean ∆ = −0.02 pp.Source: §5 · pp. 6–7
  • Reading diverged text is not harder for probes: mean accuracy differences across trajectory groups change sign and never exceed 0.58 pp; identical-trajectory accuracy falls to 87.7% at token 20.The passage links larger decode-position changes to smaller margins together with differing inputs.Source: §5 · p. 7
  • At token 20, accuracy moves 0.31 pp while 2.76% of verdicts change; prompt verdict flips are 0.076%, despite accuracy changes never exceeding 0.47 pp.Across prompt transfers, 74% leave accuracy unchanged and mean |∆| is 0.04 pp.Source: §5 · pp. 6, 8, Table 1 · p. 7
  • Same-token rows flip at 0.12–0.15% across decode positions, versus 12.9% for diverged rows; the diverged fraction predicts |∆| with Spearman ρ = 0.73.The association remains in the same-GPU subset (ρ = 0.76) and within bf16 and fp16 (ρ = 0.53 and 0.54).Source: §5 · pp. 7–8, Figure 2 · p. 7
  • At token 20, Cohen’s κ is 0.941 and AUROC moves by at most 0.05 pp; under fp32 batch-size changes, none of 201,960 verdicts flip.Positive and negative rows have similar flip rates: 2.65% and 2.84%, respectively.Source: §5 · p. 8
  • A bf16 batch-size change flips 0.14% of probe verdicts but 2.1% of the model’s own next-token argmaxes, a 15× difference.The passage attributes this contrast to a binary probe boundary versus near-ties among the model’s 105-token argmax candidates.Source: §5 · p. 8
  • Same-GPU transfers alone support the conclusions, but this design does not identify the GPU’s independent contribution.Prompt accuracy changes by at most 0.47 pp for same-GPU and 0.38 pp for cross-GPU transfers; their composition differs.Source: §5 · p. 8

6 DISCUSSION

Prompt-reading probes can transfer across serving configurations with little accuracy change, while generated-token monitors face configuration-dependent text. Aggregate accuracy hides verdict churn, and the study’s deployment guidance is bounded by its tested settings.

6 DISCUSSION

  • At the prompt, the worst observed accuracy change was 0.47 pp; generated-token deployments should use batch-invariant decoding, deterministic re-encoding, or measure their own flip rate.For auditing, fp32 was exactly reproducible across batch sizes in prompt activations, realised tokens, and probe verdicts.Source: §6 · p. 8
  • Accuracy understates verdict churn by 2–9×, so monitors judged on accuracy alone can appear stable while changing many individual decisions.The paper distinguishes benign prompt perturbations, behaviorally consequential decode changes, and validity failures.Source: §6 · pp. 8–9
  • The results are bounded by batch sizes 4–16, three dense 4B–8B models, two binary tasks, and test-split measurements at the final layer for an fp32-trained probe.For Llama and Qwen, fp32 runs used a different GPU, so precision and GPU effects cannot be separated; fp32 batch invariance applies only to the tested GPU and kernels.Source: §6 · p. 9

7 CONCLUSION

The conclusion finds that batch size and precision perturb activations, but prompt probes remain stable while decode-time token changes can alter probe verdicts. The paper recommends evaluating per-example agreement rather than relying on aggregate accuracy.

7 CONCLUSION

  • Under half a point of prompt accuracy change and fewer than one verdict flip in a thousand contrast with decode-time token changes, which coincide with most probe verdict flips.Aggregate accuracy hid verdict churn by two to nine times.Source: §7 · p. 9

AI USE STATEMENT

  • Generative AI tools assisted with drafting, editing, analysis and plotting scripts, and literature searches; the authors checked the experimental design, data collection, and scientific claims.The authors take full responsibility for the paper’s content.Source: AI USE STATEMENT · p. 9

ETHICS STATEMENT

  • The study uses public models and datasets, involves no human subjects, and reports no direct negative use; its findings may help practitioners avoid misjudging monitor stability.Source: ETHICS STATEMENT · p. 9

REPRODUCIBILITY STATEMENT

  • The authors state that they will release extraction, probe-training, and analysis code, per-row metric caches, and scripts for generating every table and figure.Runs use seed 42 and deterministic PyTorch algorithms, with dataset manifests pinned to fixed revisions.Source: REPRODUCIBILITY STATEMENT · p. 9

A METRIC DETAILS

The appendix defines metrics for element-level numerical differences, batch-size distance behavior, layerwise amplification, representation geometry, and dependence on input magnitude and sequence length.

A METRIC DETAILS

  • For fp32 arrays, ULP distance counts representable fp32 values between elements; upcast bf16 and fp16 steps span 216 and 213 fp32 steps, respectively, at matching exponents.Source: A METRIC DETAILS · p. 13
  • The triangle test compares d(4, 16) with additive and common-mean predictions from d(4, 8) and d(8, 16), then reports which prediction is closer.Source: A METRIC DETAILS · p. 13
  • Per-block amplification is the distance ratio between consecutive recorded blocks, raised to the inverse block gap, with block 3 used as the floor.Source: A METRIC DETAILS · p. 13
  • Geometry metrics include linear CKA, Procrustes residual, 10-nearest-neighbor Jaccard overlap, and Spearman correlation between data and perturbation variance along the top 50 principal components.The metrics use centred matrices, with up to 3,000 rows for most measures and 1,500 for kNN overlap.Source: A METRIC DETAILS · p. 13
  • Input dependence is assessed by OLS regression of per-row relative ℓ2 distance on activation norm and sequence length.Source: A METRIC DETAILS · p. 13

B BIMODALITY OF DECODE-POSITION DIVERGENCE

At generated token 20, decode-position relative activation distances are bimodal for most rows under bf16 batch-size comparisons, while the selected divergence threshold lies in the distribution’s valley.

B BIMODALITY OF DECODE-POSITION DIVERGENCE

  • At token 20, 74% of rows have relative ℓ2 distances in [10^-4.5, 10^-1.5), while 24% lie in [10^-0.5, 10^0.5); thresholds in [10^-1.5, 10^-0.5] shift the pooled diverged fraction by at most 1.5 pp.Only 1.5% of rows fall in the valley between the two modes.Source: B BIMODALITY OF DECODE-POSITION… · p. 13
  • Figure 3 compares activation-distance criteria with token-ID measurements at generated token 20, with one point per model, dataset, and perturbation and a dotted equality line.Source: Figure 3 · p. 13

C WHAT THE STORED DATA CAN STILL ANSWER

Stored logits support comparisons of token agreement and probe verdicts across identical and diverged trajectories, although only positions 5, 10, and 20 were logged. Decode-position accuracy gaps are centered on zero, while token-20 verdict flips are much more common among rows with diverged text.

C WHAT THE STORED DATA CAN STILL ANSWER

  • Stored logits yield per-position token agreement, the first disagreement position, and probe comparisons on identical- versus diverged-trajectory subsets.Only positions 5, 10, and 20 were logged, limiting the resolution of trajectory reconstruction.Source: C WHAT THE STORED DATA CAN STILL ANSWER · p. 14
  • Decode-position accuracy gaps are centered on zero at every position, attributing transfer changes to differing inputs rather than probe mishandling of diverged text.The comparison evaluates probes on activations produced by tokens the other run never generated versus agreed tokens.Source: C WHAT THE STORED DATA CAN STILL ANSWER · p. 14

D VERDICT FLIPS BY PERTURBATION

  • At token 20, verdict flips among diverged rows remain 12.5–15.8% across perturbations, so perturbations mainly affect how often text changes.Table 2 pools models, datasets, and probe types; fp32 batch-size pairs flip no verdicts.Source: Table 2 · p. 14

E TOKEN DIVERGENCE IN FULL

  • Table 3 reports token disagreement by model–dataset pair and probe accuracy gaps; its per-cell gaps scatter both ways and have wide sampling error.The pooled estimate, centered on zero at every position, is the one to read; Gemma-3-4B has no fp16 comparisons.Source: Table 3 · p. 15

F ACTIVATION DIVERGENCE IN FULL

  • Table 5 defines decode divergence as rows with block-3 relative ℓ2 above 0.1 at token 20 and reports prefill medians across layers, datasets, and pairs.Its GPU column identifies comparisons that change hardware and precision together; one native bf16 step equals 216 fp32 ULPs and one fp16 step equals 213.Source: Table 5 · p. 16

G PROBE TRANSFER IN FULL

Probe transfer is evaluated as accuracy change from a matched configuration, pooled across models, datasets, layers, and probe types. Secondary-factor results show modest architecture effects, greater sensitivity for the harder task, and more variation by position than depth.

G PROBE TRANSFER IN FULL

  • Probe architecture has a modest decode-position effect: accuracy changes in 78% of LR cells, versus 71% for MLP-1 and 74% for MLP-3.Mean |∆| is 0.33 pp for LR, 0.24 pp for MLP-1, and 0.28 pp for MLP-3; capacity neither buys nor costs robustness here.Source: G PROBE TRANSFER IN FULL · p. 16
  • Decode transfer is more sensitive on Rotten Tomatoes than SMS Spam, with mean |∆| of 0.36 pp versus 0.20 pp.Depth ranges from 0.18–0.28 pp, while position ranges from 0.04–0.37 pp.Source: G PROBE TRANSFER IN FULL · p. 16

H TRANSFERS SPLIT BY EXTRACTION GPU

  • Table 6 defines ∆ as test-accuracy change from a matched configuration, pooled across models, datasets, layers, and probe types; training batch size is 4.Table 7 separates transfers by whether training and test activations were extracted on the same GPU; no batch-size-only transfer changes GPU.Source: Table 6 · p. 17, Table 7 · p. 18

I REPRESENTATIONAL GEOMETRY

Representational geometry remains highly similar at the prompt but can diverge during decoding, especially in small neighborhoods. Matched-configuration probe accuracy is highest at prefill and declines with generation step.

I REPRESENTATIONAL GEOMETRY

  • Prompt Procrustes residual is 1.6×10^-2 for bf16 batch-size changes and 1.8×10^-3 for fp16, a ratio of 9.Residuals grow mildly with depth rather than showing geometric growth from per-block amplification.Source: I REPRESENTATIONAL GEOMETRY · p. 18
  • For bf16 batch-size changes, prompt 5-nearest-neighbor overlap is 94.2%, versus 95.9% for 50-nearest neighbors; at token 20 the figures are 43.5% versus 50.6%.Rounding noise reorders near-ties, affecting tight neighborhoods first; k-NN readouts are more exposed than the linear and MLP probes studied.Source: I REPRESENTATIONAL GEOMETRY · p. 18
  • At token 20, linear CKA falls to 0.64 for bf16 batch-size changes and 0.95 for fp16 batch-size changes, after staying ≥0.995 at the prompt.The decline follows generated-token index and mirrors the trajectory-divergence curve.Source: I REPRESENTATIONAL GEOMETRY · p. 18, Figure 6 · p. 19
  • The two runs’ participation ratios agree to a median ratio of 1.000, with a 5th–95th percentile range of 0.97–1.03, even when CKA falls below 1.The perturbation moves points within a cloud of unchanged effective dimension.Source: I REPRESENTATIONAL GEOMETRY · p. 18

J MATCHED PROBE ACCURACY

  • Matched-configuration accuracy is highest at prefill and declines as the probed token moves away from the input; Rotten Tomatoes’ middle, three-quarter, and final blocks reach 83.1–84.7% pooled across positions.At the prompt those blocks reach 89.7–90.0%, while block 3 reaches 75.1%.Source: J MATCHED PROBE ACCURACY · p. 19, Table 8 · p. 20

Related papers

Loading 2609.31796v1…