PaperlayerView exact revision on arXiv

Source-linked AI summary

Lost in the bf16 Cast: Exporting Ternary Language Models Can Revert Most Low-Learning-Rate Code Changes

Avichal Sahai, Nishant Raj, Animesh Srivastava

arXiv:2610.07853v1cs.LGcs.CL

TL;DR

Ternary-model fine-tuning updates higher-precision latents, but documented bf16 export can leave deployed codes different from those used during evaluation. The paper audits three labs’ pipelines and tests ways to preserve online codes through export. At the tested endpoints, documented export sharply lowers strict-format accuracy in two models, while two remedies meet the stated non-inferiority criterion in all three.

  • Problem

    Ternary models are fine-tuned with higher-precision latent weights but deployed through exports that cast latents to bf16, raising the question of whether deployed codes preserve fine-tuned changes.

    Source: §1 · p. 1

  • Method

    The paper audits documented export pipelines across three labs, compares deployed codes with fp32 quantization of released latents, evaluates fine-tuned endpoints, and tests code-writing and input-projection remedies.

    Source: §1 · pp. 1–2

  • Results

    Documented export lowers greedy GSM8K strict accuracy from 58.79% to 0.78% for Falcon-E-1B-Base and from 36.13% to 0.39% for BitCPM-CANN-0.5B; two remedies meet a 4-point strict-accuracy non-inferiority criterion in all three models.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The tested remedies preserve compatibility with online evaluation by writing the training quantizer’s codes directly or adjusting bf16 inputs until unchanged tools emit them.

    Source: §1 · p. 2

    Fine-tuned endpoint results cover one recipe family, one or two trajectories per arm, and 512 GSM8K dev prompts under one greedy decoding configuration.

    Source: §9 · p. 8

Abstract

from arXiv · show

Ternary language models such as BitNet b1.58, Falcon-E and BitCPM are fine-tuned with higher-precision latent weights and deployed as ternary codes produced by an export step that, in the labs' documented pipelines, first casts the latents to bf16. We audit those pipelines across three labs. In released checkpoints, fp32 quantization of the shipped latents disagrees with the deployed codes on 0.83-1.77% of codes in Falcon-E and BitCPM and on 1.530% in BitNet 2B-4T; for Falcon-E and BitCPM most disagreements are products that bf16 rounding lands exactly on the threshold, which ties-to-even maps to zero, and the unmodified onebitllms exporter reproduces all four Falcon-E releases byte for byte. At fine-tuned endpoints, with learning rates selected to match a nominal learning-rate-to-bf16-ULP ratio, the documented export lowers greedy GSM8K strict accuracy from 58.79% to 0.78% for Falcon-E-1B-Base and from 36.13% to 0.39% for BitCPM-CANN-0.5B, and a bf16 save and reload lowers BitNet 2B-4T's strict accuracy by 27.54 points while its last-number accuracy rises. Two compatibility remedies, writing the training quantizer's codes directly or adjusting the bf16 inputs until the unchanged tools emit them, each met a 4-point strict-accuracy non-inferiority criterion against online evaluation in all three models. In two model families, randomized interventions on the initial distance from the threshold support distance-dependent selection of the codes that fine-tuning changes.

1 INTRODUCTION

Ternary fine-tuning updates higher-precision latents, but documented bf16 export can return just-crossed weights to their starting values, so the evaluated and deployed models may differ. The paper audits this effect across three labs, measures endpoint accuracy, and tests compatibility remedies.

1 INTRODUCTION

  • Fp32 quantization of released latents disagrees with deployed codes on 0.83–1.77% of Falcon-E and BitCPM codes and 1.530% of BitNet 2B-4T codes.For Falcon-E and BitCPM, most disagreements are bf16 products exactly at the threshold, which ties-to-even maps to zero; unmodified onebitllms reproduces all four Falcon-E releases byte for byte.Source: §1 · p. 1
  • At matched nominal learning-rate-to-bf16-ULP ratios, documented export drops greedy GSM8K strict accuracy from 58.79% to 0.78% (Falcon-E-1B-Base) and 36.13% to 0.39% (BitCPM-CANN-0.5B).A bf16 save and reload lowers BitNet 2B-4T strict accuracy by 27.54 points while its last-number accuracy rises by 7.62.Source: §1 · p. 1
  • Writing the training quantizer’s codes directly or adjusting bf16 inputs until unchanged tools emit them met a 4-point strict-accuracy non-inferiority criterion against online evaluation in all three models.The two approaches are presented as compatibility remedies for preserving the online codes through export.Source: §1 · pp. 1–2
  • Randomized interventions that change only initial threshold distance support distance-dependent selection of the codes fine-tuning changes in two model families.On 2B-4T, changed codes are increasingly enriched for positive first-order base-gradient scores as initial distance from the threshold increases.Source: §1 · p. 2
  • The results cover one recipe family, one or two trajectories per arm, and 512 GSM8K dev prompts; the authors claim neither a new rounding mechanism nor a general export penalty.For prospectively packaged experiments, the analysis and decision rule were frozen before the run, with exceptions described in Appendix A.Source: §1 · p. 2

2 BACKGROUND

Ternary quantization maps latent weights to −1, 0, or +1 codes using a tensor- or row-specific scale, while quantization-aware training updates higher-precision latents. The background explains how bf16 rounding can undo a code change and where bf16 enters the documented exports.

2 BACKGROUND

  • Absmean quantization maps latent weights to q = round(clamp(Wr, −1, 1)) and deploys q · a, with a = mean|W| and r = 1/a, per tensor or per row.A weight is nonzero when |w| exceeds a/2, equivalently |w · r| > 1; BitCPM uses per-row scaling.Source: §2 · p. 2
  • Quantization-aware training uses codes in the forward pass and updates fp32 latent weights in the backward pass through a straight-through estimator.Export operation order and bf16 roundings are part of what the release audit measures.Source: §2 · p. 2
  • A latent can cross the quantization threshold yet revert if its endpoint remains inside its starting bf16 rounding cell; exact midpoint ties go to the even neighbor.Near the threshold, bf16 spacing is 0.4%–0.8% of the threshold; the export’s recomputed scale can also move the actual threshold.Source: §2 · p. 2
  • All three documented export paths cast latents through bf16, while two also compute the scale and products in bf16.Table 2 compares fp32 codes from released latents with deployed codes; release arithmetic reproduces Falcon-E and BitCPM codes exactly, but no tested bf16 recipe does so for 2B-4T.Source: §2 · p. 2, Table 2 · p. 2

3 RELEASED CHECKPOINTS ENCODE A DIFFERENT MODEL

The release audit compares deployed codes with codes obtained by fp32 quantization of matched released latents, then examines where disagreements arise and what they mean for measured loss. The evidence differs by model family and does not establish a single cross-release loss direction.

3 RELEASED CHECKPOINTS ENCODE A DIFFERENT MODEL

  • For Falcon-E and BitCPM, 93.52%–95.71% of disagreements in the reported cases are bf16 products exactly at |w · r| = 0.5, which torch.round maps to zero.The remainder are products just below 0.5 where the bf16 scale moved the threshold; ties-away rounding changes the dominant error direction without removing the precision-induced disagreement.Source: §3 · p. 2
  • The unmodified onebitllms 0.0.4 quantize-to-1bit exporter reproduces all four Falcon-E releases byte for byte, including packed codes, weight scales, and non-projection tensors.This establishes computational provenance, not a recovered export log.Source: §3 · pp. 2–3
  • The fixed-threshold schematic shows that crossing the quantization threshold is insufficient for a lasting code change when the endpoint remains inside the starting bf16 rounding cell.The example starts 0.3 ULP below threshold and moves 0.4 ULP, ending 0.1 ULP above threshold but below the rounding midpoint.Source: Figure 2 · p. 3
  • 2B-4T has 31,895,216 code disagreements, all within 0.5 ULP of the threshold in both directions; its original master and export history are unavailable.About 41% of code changes made by fine-tuning from its bf16 master land on these coordinates and afterwards agree with the packed code; the pattern is compatible with rounding, not restoration.Source: §3 · p. 3
  • On C4, deployed codes raise cross-entropy over fp32 quantization by +0.0125 nats per token for Falcon-E-1B-Base and +0.0077 for 3B-Base, lower it by −0.0053 for 2B-4T, and show no detectable difference for two Instruct releases.These are declared comparisons on validation shard 00001 using 1,024 documents and one common forward with int8 activations.Source: §3 · p. 3
  • In a separate precision contrast on the same BitCPM-CANN latents, bf16 output has lower loss than the fp32 counterfactual by −0.035544 for 0.5B and −0.006814 for 3B.Rounding ties away from zero is worse than both; the separate comparisons measure different estimands, so the paper infers no rule across releases.Source: §3 · p. 3

4 THE EXPORT AT FINE-TUNED ENDPOINTS

At low-rate fine-tuned endpoints, documented exports undo most code changes and can sharply reduce strict-format accuracy; effects depend on learning rate, export path, and scoring rule.

4 THE EXPORT AT FINE-TUNED ENDPOINTS

  • The comparison used 300-step GSM8K fine-tuning and 512 greedy-decoded dev prompts; the nominal learning-rate-to-bf16-ULP ratio chose rates but did not match training dynamics.Falcon-E's documented exporter was exercised, but its training stack was not fully replicated.Source: §4 · pp. 3–4
  • 58.79% to 0.78%: Falcon-E-1B-Base greedy strict accuracy fell after export, while answer-format adherence dropped from 97.1% to 21.7%.Last-number accuracy fell from 58.79% to 47.07%, but the exported model remained 23.24 points above the base model under that rule.Source: §4 · p. 4
  • 91.2% of Falcon-E-1B-Base code additions lay within 0.25 bf16 ULP of threshold, and the cast alone reverted 96.3% of them.The documented export raised dev cross-entropy by +0.41513 nats per token; the projection-latent cast contributed the largest ordered-path increment, +0.38053 [+0.36875, +0.39357].Source: §4 · p. 4
  • Export-loss point estimates decreased from +0.09436 at 6.25×10^-6 to +0.01524 at 10^-5 and +0.00264 at 2 × 10^-5, but the tested sequence was nonmonotonic.At 5 × 10^-5 export slightly lowered loss (−0.00147 [−0.00256, −0.00034]); at the documented 5 × 10^-4 rate the contrast was near zero (+0.00011 [−0.00041, +0.00065]), with no between-rate difference tested.Source: §4 · p. 4
  • In the low-rate Falcon-E factorial, reverted additions and newly zeroed codes interacted strongly, while all four conditional amplitude contrasts included zero.The authors caution that F1 is not the cast, F2 is not the arithmetic, and neither factorial contrasts nor ordered-path increments are shares of the cost.Source: §4 · p. 4
  • 27.54 points: BitNet 2B-4T strict accuracy fell after bf16 save and reload, while last-number accuracy rose by 7.62 points.The result concerns retained endpoints; no retraining occurred for the declared evaluation retry.Source: §4 · p. 4
  • 95.3% of BitCPM-CANN-0.5B's 1,122,949 changed codes reverted to their start code after the low-rate bf16 cast, and both scoring rules lost accuracy.At 32 times the rate, strict accuracy showed no detectable change and the cast reverted 4.0% of 14,972,249 changes.Source: §4 · p. 4
  • At these low-rate endpoints, documented exports undo most fine-tuning code changes and can collapse strict-format accuracy, with effects depending on learning rate, export path, and scoring rule.The evaluated rows come from the training split and were used in earlier analyses, not a fresh test set.Source: §4 · p. 4

5 TWO REMEDIES

The two remedies reproduce the online codes and meet the predeclared 4-point strict-accuracy non-inferiority rule against online evaluation. That result does not establish zero loss, identical answers, or scoring-rule-independent accuracy restoration.

5 TWO REMEDIES

  • The δ = 4-point margin reflects the documented contrasts’ resolution, not an application tolerance.It is the median 95% half-width of the documented-export primaries, rounded down and fixed before launch.Source: §5 · p. 5
  • Every remedy interval’s lower bound exceeds −4 points, meeting the predeclared non-inferiority rule without guaranteeing zero loss.The intervals still include losses of up to 1.95 and 2.73 points under fixed export and 1.17, 2.54, and 1.37 points under projection.Source: §5 · p. 5, Table 4 · p. 6
  • Both remedies reproduce online codes at every coordinate, but generations match online on only 395, 266, and 314 of 512 rows under fixed export and 390, 281, and 305 under projection.Once codes agree, exporter scale representation, inference-path arithmetic, and casting of non-projection parameters remain in the complete exported path.Source: §5 · p. 5
  • Projection changes 0.629% of projection coordinates by 1–3 ULPs, correcting every code the documented exports got wrong at these endpoints.Median relative L2 distance to the fp32 latents rises from 2.97, 3.01, and 2.53×10−4 for the plain cast to 4.34, 4.25, and 4.03×10−4; global minimality is not established.Source: §5 · p. 6
  • The fixed export writes captured online codes into the deployed representation, whereas projection adjusts bf16 inputs while keeping each lab’s tool unchanged.Both start from one fine-tuned endpoint per model and compare deployed and online states on the same 512 dev rows.Source: §5 · pp. 4, 6

6 WHICH CODES FINE-TUNING CHANGES

The randomized initial-distance interventions support distance-dependent selection of code changes in two model families, while the observed 2B-4T pattern also shows greater enrichment of positive first-order scores at larger initial distances. The evidence is conditional on the tested strata and interventions, and does not establish benefits for individual coordinates or improved task performance.

6 WHICH CODES FINE-TUNING CHANGES

  • On 2B-4T GSM8K, 12,905,594 of 2,084,044,800 codes change, with 99.97% at the stored level nearest the threshold.Same-level lift compares crossers’ share of positive first-order scores with that of eligible coordinates at the same level; positive scores predict lower loss.Source: §6 · p. 6
  • On 2B-4T, same-level lift rises from +0.063 below 0.1 ULP to +0.157 at 0.5–1 ULP.Figure 4 plots lift by initial distance at GSM8K endpoints, a second seed, BoolQ, and MATH; only qualifying bins are drawn.Source: §6 · pp. 6–7, Figure 4 · p. 6
  • Far-moved coordinates’ lift exceeds near-moved coordinates’ in both model families, supporting distance-dependent selection under the joint intervention.Moving coordinates farther also makes them less likely to cross; on Falcon-E the far-minus-near crossing probability is −0.133 [−0.147, −0.119].Source: §6 · p. 7, Table 6 · p. 7
  • The intervention evidence excludes strata failing the far-role 20-crosser floor, about 34% of pair weight; Falcon-E support is concentrated at smaller initial distances.Falcon-E’s supported strata average 0.194 ULP initial distance versus 0.411 ULP for excluded strata, and its estimate fell below the frozen prediction.Source: §6 · p. 7
  • Reverting far-role crossers raises dev loss by +0.00325 [+0.00258, +0.00391] nats per token more than reverting an equal number of near-role crossers.The joint intervention does not establish a benefit for individual coordinates or improved task performance.Source: §6 · p. 7

7 IMPLICATIONS FOR DEPLOYING TERNARY FINE-TUNES

Deployment guidance is to preserve and verify the training quantizer’s codes, and to score more than one way; reversion is expected when changed latents remain within their starting bf16 rounding cells.

7 IMPLICATIONS FOR DEPLOYING TERNARY FINE-TUNES

  • Preserve the codes produced by the training quantizer and verify deployed codes against online codes before task evaluation.A CPU census takes seconds and directly measures export code changes.Source: §7 · p. 7
  • A single scoring metric can miss format-related harm: 2B-4T’s harm was mostly to answer format, while Falcon-E’s and BitCPM’s appeared under both scoring rules.Source: §7 · p. 8
  • Reversion is expected when fine-tuning leaves changed latents inside their starting bf16 rounding cells; low learning rates make this likely, but long runs can also end near a boundary.The paper did not test RL fine-tuning of ternary models, so its exposure to export reversion remains a hypothesis.Source: §7 · p. 8

8 RELATED WORK

The related work connects bf16 rounding and requantization to prior precision concerns, distinguishes this audit from earlier release reports, and contrasts its controlled threshold-distance measurements with prior approaches.

Precision and requantization

  • Nearest rounding in bf16 can cancel small weight updates (Zamirai et al., 2020), while prior work addresses requantization, deployment gaps, adapter loss, and quantizer-aware export.The paper frames its fixed export as a ternary instance of QATFactory’s inference-engine numerical-specification principle (Xu et al., 2026b).Source: §8 · p. 8

Release artifacts

  • Earlier reports noted disagreement between 2B-4T’s bf16 master and packed checkpoint in sampled tensors; this paper provides a full census and fine-tuning consequence.The authors found no report of bf16 export reverting fine-tuning of these models in their bounded search.Source: §8 · p. 8

Threshold distance

  • Prior threshold-distance work includes TerMeZO’s focus on latents closest to decision boundaries (Sifaou et al., 2026); this paper instead measures crossers against same-level pools and intervenes on starting distance.The authors describe the TerMeZO comparison as a contrast in design premise, not a measured contradiction.Source: §8 · p. 8

9 LIMITATIONS

The evidence is bounded by a narrow fine-tuning and evaluation setup, specific software paths, and limited endpoint and threshold-distance experiments.

9 LIMITATIONS

  • Endpoint results use one recipe family, one or two trajectories per arm, and 512 GSM8K dev prompts from the training split under one greedy decoding configuration.Greedy answers are sensitive to batch configuration.Source: §9 · p. 8
  • The study measures PyTorch bf16 paths, not bitnet.cpp, CANN, or integer kernels; Falcon-E training uses Transformers’ AutoBitLinear rather than TII’s Triton BitNetLinear.Remedies use one existing endpoint per model and one attempt each; non-inferiority at a resolution-based margin is not equivalence, and projection is limited to these tool versions.Source: §9 · p. 8
  • The threshold-distance study examines one released model in depth, with one randomized intervention per model family, first-order scores, and an outcome-dependent support rule.Source: §9 · p. 8
  • Two BitCPM evaluations shared a GPU with a production job; phase-boundary checks and sampled occupancy logs cannot prove exclusivity.The literature search was bounded, and the authors make no priority claim.Source: §9 · p. 8

10 CONCLUSION

The conclusion is that documented bf16 exports can revert many low-learning-rate code changes and collapse strict-format accuracy at tested endpoints, while two compatibility remedies avoided that collapse on these rows.

10 CONCLUSION

  • Documented bf16 exports can silently revert most low-learning-rate fine-tuning code changes and collapse strict-format accuracy at the tested endpoints.The accuracy effect depends on scoring rule; 2B-4T’s last-number accuracy rose.Source: §10 · p. 8
  • Writing the training quantizer’s codes directly or adjusting bf16 inputs so unchanged tools emit them avoided the strict-accuracy collapse and met the declared four-point non-inferiority rule on these rows.Source: §10 · p. 8
  • Ternary fine-tuning pipelines should export with the quantizer used for training and verify deployed codes against online codes.Source: §10 · pp. 8–9

STATEMENT ON AI ASSISTANCE

The project used AI assistants for analysis, drafting, and review, while preserving evidence records and distinguishing independent AI checks from human peer review. The appendix documents experiment provenance and release-audit comparisons that separate code and scale effects from arithmetic choices.

STATEMENT ON AI ASSISTANCE

  • AI assistants supported analysis, drafting, and review; the separate “independent review” was an AI consistency check, not human peer review.Authors retain responsibility for all claims, and later result reviews recomputed primary contrasts from saved records.Source: STATEMENT ON AI ASSISTANCE · p. 9, A PROTOCOL AND PROVENANCE · p. 11

A PROTOCOL AND PROVENANCE

  • Experiments were prospectively preregistered, packaged with hashed inputs and frozen analysis code, then checked at phase boundaries before launch.GPU time was recorded by phase; appendix numbers were generated from the manuscript and checked against accepted-run evidence files.Source: A PROTOCOL AND PROVENANCE · p. 11
  • The GSM8K threshold-distance association was found exploratorily, while BoolQ and MATH were declared prospectively; Experiments 12 and 18 were adaptive.Experiment 13 failed before Experiment 25 was declared, and Experiment 25’s prediction estimator was chosen from Experiment 13’s data. Experiment 9 was accepted as qualified pilot evidence; its input bindings were verified after its verdict, and its probe identity is unverified.Source: A PROTOCOL AND PROVENANCE · p. 11
  • Experiment 22 reran evaluation on Experiment 21’s retained endpoints after an out-of-memory defect blocked the earlier run; no retraining occurred.Source: A PROTOCOL AND PROVENANCE · p. 11
  • GPU availability checks were not continuous, and production jobs shared the GPU during parts of B2’s Experiment 23 evaluation and BitCPM’s Experiment 26 fixed-state evaluation.Experiment 26 also launched within a window excluded by its pre-launch review.Source: A PROTOCOL AND PROVENANCE · p. 11

B ADDITIONAL RELEASE-AUDIT RESULTS

  • Table 7 decomposes the Falcon-E comparison into code, scale, and interaction contributions, while its intervals are correlated and a nondetectable difference is not equivalence.A is fp32 quantization of released latents; B is deployed codes and scales; C and D swap codes and scale, respectively.Source: B ADDITIONAL RELEASE-AUDIT RESULTS · p. 11, Table 7 · p. 12
  • Table 8 contrasts BitCPM’s release bf16 arithmetic with an fp32-arithmetic counterfactual and separates code, magnitude, and tie-rounding choices.The release’s bf16 arithmetic on released latents reproduces deployed weights; E rounds ties away from zero at the bf16 product and magnitude.Source: B ADDITIONAL RELEASE-AUDIT RESULTS · p. 12, Table 8 · p. 12

C ADDITIONAL FINE-TUNED-ENDPOINT RESULTS

The appendix details fine-tuned endpoint comparisons and defines the threshold-distance analyses. It specifies how export stages, code additions, distance contrasts, and exact-partition interactions are measured.

C ADDITIONAL FINE-TUNED-ENDPOINT RESULTS

  • The declared path separates online evaluation, cast to Ptrained, export of Ptrained, and export into three successive steps: projection cast, export arithmetic, and non-projection cast.Arm labels are local to each experiment, and Table 9 reports nats per target token with row-bootstrap intervals.Source: C ADDITIONAL FINE-TUNED-ENDPOINT RESULTS · p. 12, Table 9 · p. 12
  • Table 10’s near share counts additions within 0.25 bf16 ULP above the fp32 threshold, including boundary cases; reverted shares count additions returned to the base code by cast and export.These are descriptive post hoc summaries, and continuous margins cannot be independently recomputed without the unsaved fp32 masters.Source: Table 10 · p. 12
  • Tables 11–14 cover Experiment 19’s nats-per-target-token results, Experiment 20’s greedy accuracy, Experiment 22’s online-versus-reloaded accuracy and dev CE, and Experiment 23’s online-versus-deployed accuracy and dev CE.Table 14 also reports the share of fine-tuning code changes returned to the start code by bf16 cast and served-model differences from the published checkpoint.Source: Table 11 · p. 13, Table 12 · p. 13, Table 13 · p. 13

D THRESHOLD-DISTANCE SELECTION IN FULL

Related papers

Loading 2610.07853v1…