PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Text, Different Prediction: Serving-Context Nondeterminism in Text Classifiers

Santhosh Kumar Kasa, Siva Rajesh Kasa, Sumit Negi

arXiv:2610.09111v1cs.CLcs.LG

TL;DR

Research has documented serving-context nondeterminism in text generation, but its effects on text classification have received little direct measurement. The paper studies controlled context changes across 180 classifiers and finds that stable labels can coexist with score movement, while structural input changes and numerical execution changes affect predictions differently.

  • Problem

    Prior work documents serving-context changes in text generation, while whether and when the same factors affect a classifier’s single forward pass remains largely unmeasured.

    Source: §1 · pp. 1–2

  • Method

    The authors evaluate 180 classifiers across serving contexts while holding the checkpoint and text fixed, and distinguish structural changes to represented inputs from numerical changes to execution.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Structural changes to represented inputs move labels across a wide range of confidence, while execution-only changes move labels near decision boundaries; stable labels can coexist with score movement.

    Source: §5 · p. 5, §6 · p. 9

  • Takeaways & Limitations

    Reproducible classification requires fixing identified serving settings, monitoring class probabilities rather than labels, and recomputing inputs on which fixed settings disagree.

    Source: §6 · p. 10

    The magnitude of effects remains to be measured for in-context classifiers, multimodal inputs, and production engines.

    Source: §6 · p. 9

Abstract

from arXiv · show

Deterministic inference is essential for reliable and trustworthy machine learning. Prior studies of text generation have shown that changing factors such as batch size, batch composition, hardware, or inference engine can alter the generated text, even when the prompt, model parameters, and sampling randomness are fixed. These differences have been attributed in part to floating-point non-associativity, shape-dependent kernel selection, and other implementation-level differences in numerical execution. However, it remains unclear whether, when, and to what extent the same factors affect text classification. We present a systematic study of serving-context non-invariance in text classifiers, which prior work has measured only through generated text. We train 180 models spanning discriminative, pseudo-generative, and fully generative classifier formulations and evaluate each across four categories of serving contexts, holding the checkpoint and the text fixed. Label stability does not imply score stability. Changing only the batch shape changes no labels across fp32 comparisons, yet under bf16 it moves up to 56.7 percentage points of predicted probability mass, with label changes concentrated at small margins. Fully generative classifiers change more labels than their discriminative counterparts under the same serving changes. We derive sufficient conditions for label stability under each serving change and give a separate mitigation for each mechanism. Our results identify and quantify the serving conditions that must be fixed for reproducible text classification.

1 INTRODUCTION

Prior work documents serving-context changes in text generation, but their effects on a classifier’s single forward pass have received little direct measurement. This study measures those effects across classifier formulations and settings, and examines their mechanisms and mitigations.

1 INTRODUCTION

  • Serving-context changes in text generation can alter outputs despite fixed prompts, parameters, and sampling seeds, but their effects on classification remain largely unmeasured.Numerical execution differences and inference-backend choices are among the documented causes in generation.Source: §1 · p. 1
  • The study trains 180 classifiers and evaluates them across 3,960 serving contexts while holding the checkpoint and text fixed.The models span four formulations, three sizes, five datasets, and three seeds.Source: §1 · p. 2
  • Changing padded length alone changes about 10% of predicted labels, while settings that change almost no labels can still shift class probabilities by tens of points.The authors argue that label accuracy therefore does not reveal the full instability.Source: §1 · p. 2
  • The paper reports a targeted mitigation for each measured mechanism and evaluates its invariance–cost trade-off.It also formalizes sufficient conditions for label stability and a three-way tension among inference speed, task accuracy, and serving-context invariance.Source: §1 · p. 2

2 RELATED WORK

Prior classifier comparisons focus on training objectives and model properties, while inference nondeterminism research primarily studies generation. This paper examines whether serving-context changes affect a fixed text classifier’s output.

2 RELATED WORK

  • Prior classifier comparisons study formulations, objectives, sample efficiency, robustness, calibration, and privacy, but not output changes caused by inference-time serving settings.The compared formulations include discriminative, pseudo-generative, and fully generative classifiers.Source: §2 · p. 2
  • Generation studies report output changes under altered batch size, precision, hardware, or inference engine, with numerical execution and backend choices among the documented causes.Those studies also propose batch-invariant kernels, fixed-shape scheduling, and verified recomputation of disagreeing decoded steps.Source: §2 · p. 3
  • The generation literature concerns autoregressive outputs, leaving open whether the same serving mechanisms affect a classifier’s single forward pass.Appendix D expands the discussion to prediction instability, calibration, numerical and positional mechanisms, and production-readiness testing.Source: §2 · p. 3

3 SERVING CONTEXT AND PREDICTIVE CLAIMS

The paper defines serving contexts as implementation settings that may vary while the checkpoint and text remain fixed, and compares predictions across controlled context pairs. It distinguishes changes to the represented input from changes to execution alone.

3 SERVING CONTEXT AND PREDICTIVE CLAIMS

  • A serving context records implementation settings that may vary while the trained parameters and text stay fixed; comparisons change one setting at a time.The selected pairs are controlled witnesses, not guarantees over every serving configuration, and no context distribution is assumed.Source: §3 · pp. 3–4
  • The paired flip rate measures how often the predicted label changes between a reference context and another context over jointly scored targets.It measures user-observable decision changes rather than aggregate quality.Source: §3 · p. 3
  • Score drift is measured using aligned class-probability vectors and maximum absolute logit change, including a separate view of unchanged-label targets.The passage defines total variation between probability vectors as a score-drift measure.Source: §3 · p. 3
  • The paper separates structural changes to the represented input from numerical changes to execution of a fixed represented input.Its two axes classify mechanism and setting locus; padding spans a boundary because its policy is deployment-level while dynamic width is realized per request.Source: §3 · p. 3

4 EXPERIMENTAL DESIGN

The experimental design uses controlled serving interventions to test prediction stability across models, datasets, and serving settings. Comparisons vary one recorded setting at a time, except for the combined device-and-quantization migration.

4 EXPERIMENTAL DESIGN

  • The experiments ask whether predictions persist across contexts, whether labels and scores agree on stability, which mechanism moves predictions, and how to distinguish effective mitigations.They also test against prediction collapse and ineffective comparisons.Source: §4 · p. 4
  • The study trains 180 classifiers from random initialization across four formulations, three jointly scaled sizes, five datasets, and three random seeds.The models use BERT or GPT-2 backbones, public tokenizers, and no pretrained weights.Source: §4 · p. 4
  • Serving interventions vary deployment-level or request-level settings, while the GPU-to-CPU comparison changes device and 8-bit quantization together.Padding comparisons retain the target and batch peers while changing static versus dynamic width; padding side is not randomized within a checkpoint.Source: §4 · p. 4
  • The model-size analysis retains matched family, dataset, and seed combinations only when all three sizes exceed 1.15 times chance validation accuracy.It leaves 40 matched combinations overall and 16 for the GPT-2 padding comparison.Source: §4 · p. 4

5 RESULTS

Serving-context changes can move classifier scores even when labels stay fixed; structural changes can flip labels across confidence levels, while numerical changes flip them near decision boundaries. The results cover deployment and request changes, combined migrations, targeted mitigations, and formal conditions for stability.

5.2 REQUEST-LEVEL SERVING CONTEXT WITHIN A FIXED DEPLOYMENT

  • Padding was the dominant request-level effect; about one in five comparisons changed labels in the two GPT-2 families, while right-padded BERT and masked-language-model classifiers changed none.Real-token positions shift with left padding when positions depend on tensor columns; assigning positions by real-token order removes the effect.Source: §5.2 · pp. 5–6, Figure 2 · p. 6
  • At full precision, batch-shape changes flipped no labels across 226,104 comparisons and moved probabilities by less than 10^-5; under bf16, one prediction’s probability moved by up to 56.7 percentage points.Dynamic 8-bit quantization also made batch composition matter, reaching about 3.7% of fully generative labels and score movement of 1.3 at batch size 256; target-slot changes moved GPU logits by about 10^-5 without label changes.Source: §5.2 · pp. 6–7

5.3 THE COMBINED EFFECT OF A WHOLE-DEPLOYMENT MIGRATION

  • About 0.3% of discriminative BERT labels, about 2% of each pseudo-generative formulation, and 5.6% on average (up to 39%) of fully generative GPT-2 labels changed under whole-deployment migration.The migration changes device, numerical format, and operator path together.Source: §5.3 · p. 7, Figure 3 · p. 7
  • The serving load changed about 1.6% of fully generative labels at full precision and more under 8-bit integer execution, while discriminative labels stayed near the floor.The driver was realized padded width rather than batch count; serving load and whole-deployment migration flipped similar label fractions but differed in score movement.Source: §5.3 · p. 7, Figure 3 · p. 7

5.4 MITIGATIONS AND THEIR COSTS

  • Mechanism-matched mitigations address separate effects: real-token-order positions remove padding drift, batch-invariant kernels remove batch-shape drift, and fixed offline scales decouple co-batched inputs.In this evaluation, dynamic padding processed about one quarter as many padded tokens for about two to two and a half times target throughput; kernel latency was roughly neutral at batch size 1 and about 1.3 to 1.5× standard latency at large batch.Source: §5.4 · p. 8, Table 2 · p. 8

5.5 FORMAL RESULTS

  • Width invariance in exact arithmetic follows when positions are shift-invariant, padding is masked, the readout follows the same real-token state, cache coordinates are consistent, and no other operation uses absolute columns.Learned absolute embeddings indexed by tensor column can represent different inputs at different widths; the result does not guarantee bitwise equality under finite precision.Source: §5.5 · p. 8
  • A label change requires the reference probability margin to be no greater than twice the total-variation drift, while fully generative score movement can grow with input length.The certificate preserves labels, not calibration, ranking, or thresholds, and is sufficient rather than necessary for stability.Source: §5.5 · p. 9
  • The conditional theorem says no implementation meets all three thresholds when Q < α, but the tested configurations did not require sacrificing speed, accuracy, or invariance.Its premise concerns execution models whose fast path exposes a fixed representation; no such model was identified for these classifiers.Source: §5.5 · p. 9

6 CONCLUSION

Serving settings can change a classifier’s prediction even when its checkpoint and text are fixed; the numerical effects can move probabilities without changing labels. The conclusion recommends fixing serving settings, monitoring probabilities, and recomputing inputs when fixed settings disagree.

6 CONCLUSION

  • Numerical changes can move probabilities while labels remain stable, but padding can change the input itself and still flip labels at wide margins.Order-based positions remove the observed padding effect without lowering accuracy in the experiments.Source: §6 · p. 9
  • Measurements cover transformer classifiers on unimodal text, while magnitudes for in-context classifiers, multimodal inputs, and production engines remain to be measured.The models were trained from scratch and fine-tuned in pretrained controls, using one GPU, one CPU, and two Android phones.Source: §6 · p. 9
  • Reproducible classification requires fixing identified serving settings, monitoring class probabilities rather than labels, and recomputing inputs where fixed settings disagree.Source: §6 · p. 10

B.1 IDENTICAL CONTEXTS ARE BITWISE REPEATABLE

  • Identical serving contexts reproduce bit-identical logits on every measured configuration, so reported differences are attributable to the varied setting.Repeated identical runs contribute exactly zero difference.Source: §B.1 · p. 16

B.2 POSITION-ASSIGNMENT INTERVENTION

  • Assigning positions by real-token order removes padding-induced class changes, while restoring column positions brings them back.Figure 5 accompanies this intervention.Source: §6 · p. 9, §B.2 · p. 16

B.3 HIGH CONFIDENCE DOES NOT GUARANTEE STABILITY

Higher predicted-class confidence coincides with fewer numerical label changes, but not with less score movement. Padding can still change labels at wide margins, and thresholds calibrated under one serving setting are not guaranteed under another.

B.3 HIGH CONFIDENCE DOES NOT GUARANTEE STABILITY

  • As predicted-class probability rose from 0.554 to 0.846, precision class changes fell from 3.68% to 0.568% and GPU-to-CPU changes from 3.30% to 0.666%, while precision score movement rose by 0.0118.The confidence comparison does not separate confidence from model size or training quality.Source: §B.3 · p. 16
  • At top-two logit margins of at least 4, padding changed 51 of 46,006 binary and 312 of 74,745 multiclass decisions, while both numerical comparisons changed none across 3.65 million comparisons.Numerical changes concentrate near tied leading scores; padding changes classes at wider gaps too.Source: §B.3 · p. 16
  • A probability threshold calibrated under one inference setting carries no guarantee under another.Confidence suppresses label changes from numerical effects, not the padding effect that changes the input.Source: §B.3 · p. 17

B.4 COMPUTATIONAL LENGTH AMPLIFIES NUMERICAL DRIFT

Numerical drift increases with real-token input length, while larger models show greater score movement under padding and batch-size changes even when labels hold. The mechanisms and results distinguish score stability from label stability.

B.4 COMPUTATIONAL LENGTH AMPLIFIES NUMERICAL DRIFT

  • Generative scoring accumulates per-token perturbations: the score drift bound grows as Lε and the top-two gap bound as 2Lε, versus 2ε for a single-output-position classifier.The bound limits possible drift but does not predict realized drift or label stability.Source: §B.4 · pp. 17–18
  • At padded width 256, bf16 maximum logit difference grew from 0.41 to 1.53 as IMDB length increased from 32 to 256, with flips rising from 2 to 7 of 256; fp32 stayed at 10^-4 scale with zero flips.The length trend held within a fixed margin band, and AG News reproduced the descriptive fp32/bf16 contrast.Source: §B.4 · p. 17, Figure 6 · p. 18

B.5 SCORE MOVEMENT UNDER PADDING GROWS WITH MODEL SIZE

  • Among unchanged predictions, padding-related average total variation rose from 0.0716 to 0.1441 across small to large models, while the fraction of changed classes showed no consistent size trend.Model size, width, depth, and training quality vary together, so these are descriptive associations.Source: Table 4 · p. 18, §B.5 · p. 18

B.6 BATCH SIZE MOVES SCORES EVEN WHERE PREDICTED CLASSES HOLD

  • Among unchanged predictions, batch-size average total variation rose from 2.26×10^-6 in small models to 2.32×10^-4 in large models, a paired increase of 2.29 × 10^-4.The comparisons pool more than one numerical precision and do not measure a pure fp32 trend.Source: §B.6 · p. 19

B.7 MATCHING THE POSITION RULE BETWEEN TRAINING AND INFERENCE

Position assignment must match between training and inference to preserve accuracy. Consistent use of order positions has accuracy differences within a ±2 percentage-point band and permits padding batches to their longest sequence.

B.7 MATCHING THE POSITION RULE BETWEEN TRAINING AND INFERENCE

  • Mismatched column-trained and order-served models lose up to 9.88 accuracy points, while consistent position rules differ by −0.15 and +0.32 percentage points at family level.A one-seed pilot using order positions processed 73.5% fewer padded token slots; its latency ratios are evaluation-setup measurements, not production speedups.Source: §B.7 · p. 19

B.8 PADDING ACROSS ARCHITECTURES: FURTHER DETAIL

Padding can change fine-tuned GPT-2 predictions, while position assignment determines whether the input itself changes with padded width. Numerical perturbations, by contrast, flip labels mainly for small-margin examples and can move scores without changing labels.

B.8 PADDING ACROSS ARCHITECTURES: FURTHER DETAIL

  • 40.015% mean class-change rate across 15 fine-tuned GPT-2 dataset–seed combinations shows padding instability at larger magnitude.The 95% confidence interval is 20.620 to 53.790%, and all models passed the accuracy threshold and predicted more than one class, although SST-5 models were weak.Source: §B.8 · p. 19
  • A no-position model changed no predicted classes across 256 padded-width changes, but its accuracy was 0.747 and logits still moved by at most 7.6 × 10^-6.Removing positional encoding removes the structural mechanism, not the numerical effect of execution shape.Source: §B.8 · p. 19

B.9 POSITION-ASSIGNMENT MECHANISM: FURTHER DETAIL

  • The position rule determines whether padding changes the represented input: real-token order and difference-based positional terms are padding-independent, unlike learned absolute positions indexed by tensor column.Rotary embeddings are among the listed difference-based schemes; ALiBi, T5, Shaw, Transformer-XL, and sinusoidal rows are mechanism boundaries supported by literature, not measurements in these experiments.Source: §B.9 · p. 20
  • Order-based positions reduced conventionally trained GPT-2 family-mean class-change rates from 16.25% and 17.37% to zero observed, while retraining with those positions yielded 0 changes in 25,986 comparisons.Deliberately restoring column positions brought back 5,769 changes in 25,986 comparisons; on CPU, column positions changed 23.4% of SST-2 and 7.8% of AG News labels, while order positions changed none.Source: §B.9 · p. 20

B.10 FURTHER DETAIL ON THE NUMERICAL MECHANISM

  • 30.6% of bf16 labels changed below margin 0.01, versus 0.18% at margin 0.25 or greater; the largest probability movement, 0.567, occurred in the widest-margin group without label changes.All 375 observed bf16 batch-induced flips met the necessary condition that confidence gap be no greater than twice the probability movement; the certificate bounds labels, not scores.Source: §B.10 · p. 20

B.11 QUANTIZATION LETS OTHER EXAMPLES IN THE BATCH CHANGE A PREDICTION

The appendix traces serving-context changes to distinct mechanisms, including batch-global quantization scales and shape-dependent matrix-multiplication order. Their effects range from score drift without label changes to label flips, and the mitigations are mechanism-specific.

B.11 QUANTIZATION LETS OTHER EXAMPLES IN THE BATCH CHANGE A PREDICTION

  • Dynamic int8 batch-global scales let co-batched examples change the target’s quantization grid; GPT-2 label changes rose from 2.08% at batch size 8 to 3.65% at batch size 256, while fixed scales restored bitwise equality.The maximum absolute logit movement was 1.32 at batch size 256; BERT showed at most 1 change out of 192 comparisons in any single cell.Source: §B.11 · p. 21

B.12 THE TARGET SLOT AND THE MATRIX-MULTIPLICATION REDUCTION ORDER

  • Changing a target’s batch slot alters the matrix-multiplication reduction order, producing roughly 10^-5 logit differences for GPT-2 and 10^-6 for BERT without crossing a decision boundary.The integer CPU path is bitwise identical across slots, and batch-invariant matrix-multiplication kernels remove the slot-dependent effect.Source: §B.12 · pp. 21–22

B.13 AUTOREGRESSIVE FEEDBACK AFTER THE FIRST TOKEN DISAGREEMENT

  • With bf16, batch size 1 versus 32 produced a greedy-token divergence within 4 to 14 steps in all 3 selected prompts, after which 91 to 119 of 128 tokens differed; fp32 diverged in none.The paper attributes this contrast to autoregressive feedback: later generation steps condition on the decoded prefix, unlike a single classification forward.Source: §B.13 · p. 22

B.14 MITIGATION COSTS AND BACKEND COMPARISONS

  • Mechanism-targeted mitigations restore invariance but do not cover other mechanisms; batch-invariant kernels reached bitwise equality on 64 of 64 targets while costing about 1.3× to 1.5× at large batch in precision-matched comparisons.Order-based positions also permit dynamic padding, processing 0.265× as many padded tokens for 2.04× to 2.51× measured target throughput; backend changes were label-stable but not necessarily score-stable.Source: §B.14 · p. 22, Figure 7 · p. 23

B.16 SERVING-CONTEXT EFFECTS BY MODEL SIZE

  • Across request-level factors, batch size, padded width, and within-batch rearrangements are compared by deployment; score movement under unchanged labels grows with model size, while label-change rate does not.The size association is not isolated from depth, width, head count, and training quality; dynamic-versus-static padding disagreement shrinks as batches grow and their padded widths converge.Source: §B.16 · p. 23, §B.15 · p. 23, §B.17 · p. 24

B.18 PRECISION COMPARISON INCLUDING FP16

  • fp16 preserved every fp32 label in tested cells, while fp32-to-bf16 and bf16-to-fp16 each moved 2 of 192 GPT-2 labels; device-only changes produced zero label changes in 128 comparisons per family.On the tested phones, dynamic int8 composition changes moved all 64 probability vectors but no labels; static int8 was bitwise identical under the same change.Source: §B.18 · p. 24, §B.19 · p. 25, §B.20 · pp. 25–26

B.21 DOWNSTREAM DECISIONS MOVE MORE THAN LABELS

  • Threshold actions, abstention, prediction sets, and ranking can change more often than predicted labels; a held-out canonical replay policy detected all 44 label disagreements while recomputing 22.45% of requests.The policy uses a fast bf16 batch-size-32 path and an fp32 batch-size-one canonical context; its calibration operating points were underpowered, and it does not cover complete probability vectors.Source: §B.21 · p. 26, §B.22 · p. 26, Table 13 · p. 28

B.23 OUTPUT-POSITION FAULTS THAT MIMIC PADDING INSTABILITY

  • Output-position faults can mimic padding instability: selecting a fixed physical column or pooling padding changed labels, whereas semantic-token gathers and pooling over real tokens did not.The appendix distinguishes these classification-head faults from width-dependent position assignment; the order comparison also changes peers and target slot, so it is not a pure within-batch permutation.Source: §B.23 · p. 27, §B.25 · p. 29

C IMPLEMENTATION DETAILS

This section describes the classifier implementations and their relation to prior work, and formalizes conditions governing representation shifts and label stability. It also examines how batch-global dynamic quantization can make logits depend on peer content and how a fixed peer-independent scale prevents such changes under stated execution assumptions.

C IMPLEMENTATION DETAILS

  • The implementation spans BERT and GPT-2 classifier formulations, with fixed tokenization and padding conventions, class-normalized inference scores, and a separate LoRA setup for ModernBERT and Qwen3.Primary models use hidden sizes 64, 384, and 768, depths 1, 6, and 12, and maximum sequence length 512. Primary optimization uses AdamW, learning rate 2 × 10−5, batch size 32, and early stopping; ModernBERT and Qwen3 use LoRA rank 16 and learning rate 2 × 10−4.Source: C IMPLEMENTATION DETAILS · p. 29

D EXTENDED RELATED WORK

  • Prior work studies classifier-family differences, generation-time execution changes, and prediction variation across models; this study instead varies serving context for one fixed checkpoint and input.The appendix distinguishes its tested set of outputs across serving configurations from credal prediction sets and calibration, which it does not estimate.Source: D EXTENDED RELATED WORK · pp. 30–31

E THEORETICAL RESULTS AND PROOFS

  • The appendix defines serving context by execution conditions while holding checkpoint, text, and tokenizer fixed, and uses total variation and top-two margins to formalize prediction stability.Its theoretical results address position assignment, quantization scale policy, generative-score accumulation, numerical-error propagation, and joint accuracy, speed, and invariance feasibility.Source: E THEORETICAL RESULTS AND PROOFS · p. 32

E.1 POSITION ASSIGNMENT UNDER CHANGES OF PADDED WIDTH

  • 35,591 of 176,778 comparisons changed labels for left-padded models with learned absolute positions; right-padded controls had zero changes over the same comparisons.The learned-absolute-position results averaged 20.1% changes, or 10.3% pooled across comparisons from the 158 pairs that passed the accuracy threshold.Source: §E.1 · p. 33
  • Relative-position schemes preserve positional terms under a common shift, while learned absolute embeddings generally change real-token representations; end-to-end invariance also requires consistent masking and output positions.The sufficient condition additionally requires coherent cache coordinates and no other operation dependent on absolute tensor columns; the listed mechanisms are not a complete serving-invariance guarantee.Source: §E.1 · pp. 33–34

E.2 MARGIN CERTIFICATES FOR LABEL STABILITY

  • A label is stable if its reference logit margin exceeds 2ε or its probability margin exceeds 2TV; all 375 observed reduced-precision changes satisfied m ≤ 2TV.The logit certificate uses the maximum per-class logit change ε, while the probability certificate uses total variation from the reference context; observed changes concentrated at small pre-intervention margins.Source: §E.2 · pp. 34–35

E.3 QUANTIZATION SCALE POLICY AND PEER COUPLING

  • Dynamic int8 peer-content changes moved logits by up to 0.774 for the fully generative model and about 0.09 for the discriminative model, with up to 2 of 64 label changes.These results came from one tested dynamic-quantization implementation and module set; the proposition describes possible, not inevitable, movement.Source: §E.3 · p. 35
  • With fixed target state, a positive peer-independent scale, peer-independent target computation and kernel behavior, and deterministic execution, changing peers at fixed shape cannot change the target output.The proposition does not guarantee invariance across batch shapes, kernels, hardware, precision, or scale-application order.Source: §E.3 · p. 36
  • The peer-content intervention became bitwise identical on the target with an offline fixed scale, unlike the runtime-scale condition.The static-int8 and dynamic-scale control comparison used fake-quantized linear modules with fp32 accumulation, isolating activation-scale source but not using a production integer kernel.Source: §E.3 · p. 36

E.4 FULLY GENERATIVE SCORING AND GREEDY DECODING

The section gives sufficient margin conditions for stable fully generative scores and greedy decoding, then bounds how context-specific discrepancies propagate through model depth. These bounds distinguish numerical execution changes from structural padding changes, without guaranteeing that deeper models drift more.

E.4 FULLY GENERATIVE SCORING AND GREEDY DECODING

  • A reference class-score margin greater than 2Lϵ ensures invariant fully generative predictions; class-dependent lengths replace this bound with (L_y + L_j)ϵ.The bound assumes each token contribution changes by at most ϵ; summed sequence scores have a larger admissible drift than single-position classifiers, not necessarily larger realized drift.Source: §E.4 · pp. 36–37
  • Approximately 0.447 versus 0.03125 was the maximum batch-shape logit movement for fully generative versus discriminative models under reduced precision in the tested cells.The comparison links score aggregation to the observed formulation difference; drift also increased with real-token length at fixed padded width, but length normalization or higher margins can reverse the ordering.Source: §E.4 · p. 37
  • If every step’s top-two logit margin g_t exceeds 2ϵ_t, greedy decoding selects identical tokens through step T.The certificate compares logit differences at the same prefix; after a token diverges, the same-input perturbation model no longer applies.Source: §E.4 · p. 37

E.5 PROPAGATION OF DISCREPANCIES THROUGH DEPTH

  • Layerwise discrepancy bounds separate an initial state difference from local context-specific map discrepancies propagated through later layers.The bound assumes layer maps are Lipschitz over reachable states and that same-state map discrepancies are uniformly bounded.Source: §E.5 · pp. 37–38
  • Numerical execution changes start with zero initial discrepancy, whereas structural padding that shifts learned absolute position embeddings generally creates a nonzero initial discrepancy.The observed association between nominal model size and hidden score drift does not establish that depth causes drift; contracting layers or cancellation can alter discrepancies.Source: §E.5 · p. 38

E.6 CONDITIONAL INFEASIBILITY OF ACCURACY, SPEED, AND INVARIANCE

The section gives conditional results linking fast, invariant execution to an accuracy ceiling, including exact and approximate forms. Their application to measured classifier paths depends on establishing the theorem’s information-restriction premise.

E.6 CONDITIONAL INFEASIBILITY OF ACCURACY, SPEED, AND INVARIANCE

  • If every fast, invariant implementation predicts deterministically from Q and the Bayes accuracy available from Q is below α, no implementation can meet the speed, accuracy, and invariance thresholds together.The information-restriction premise is assumed, not established for the classifier family studied.Source: §E.6 · p. 38
  • A collision graph built from inputs indistinguishable on a fast path forces any fast, label-invariant classifier to predict labels as a function of each input’s connected component Q.For exact score invariance, the full probability vector is likewise a function of Q; the collision must occur within one context, and an application must establish the lossy fast representation for the actual path.Source: §E.6 · p. 39
  • A four-input example shows each pair of accuracy, speed, and invariance thresholds can be feasible while all three are not.Its one-unit fast path and two-unit slow path are abstract costs, not measured latencies; the loss comes from overlapping collisions across contexts.Source: §E.6 · p. 40
  • The approximate bound derives disagreement ρ_f from collision structure, component diameter, score tolerance, and the reference-context margin tail, yielding an accuracy ceiling of A*_Q + ρ_f.The bound can be loose; a family-wide conclusion requires a uniform bound on ρ_f and α > A*_Q + ρ.Source: §E.6 · p. 40
  • The theorem is conditional rather than architecture-specific: the paper does not establish its information restriction for the measured GPU and CPU paths or measure a controlled joint frontier.Measured mitigation costs indicate that the fast-and-invariant subset need not be empty in tested configurations.Source: §E.6 · pp. 38–40

F EXTENDED LIMITATIONS

The limitations bound the study’s measured effects to particular models, configurations, and evaluation settings; several adjacent settings remain unmeasured.

F EXTENDED LIMITATIONS

  • The study covers transformer text classifiers under conventional fine-tuning, with LoRA controls, but not in-context instruction-tuned classification or multimodal classifiers.The latter combine text with tabular, image, or audio features.Source: F EXTENDED LIMITATIONS · p. 41
  • Hardware and backend measurements use one GPU, one CPU, two Android phones, PyTorch eager, torch.compile, and ONNX Runtime; magnitudes are specific to these configurations.Production engines, tensor parallelism, and other accelerators require their own measurements.Source: F EXTENDED LIMITATIONS · p. 41
  • Evaluation uses standard IID test splits and designed balanced grids, so measured effect frequency is not its incidence in production traffic.Behavior under distribution shift remains unmeasured.Source: F EXTENDED LIMITATIONS · p. 41
  • Architecture comparisons hold each family’s reference-code conventions fixed, and the from-scratch size series jointly scales depth, width, and head count.The size trends are conditioned on all three sizes passing the accuracy threshold.Source: F EXTENDED LIMITATIONS · p. 41
  • The pretrained replication covers one 124M-parameter GPT-2 family and formulation; its mitigation evidence comes from training and serving the same position rule.Applying that rule only at inference degrades accuracy or collapses predictions, and larger pretrained families remain to be measured.Source: F EXTENDED LIMITATIONS · p. 41

Related papers

Loading 2610.09111v1…