PaperlayerView exact revision on arXiv

Source-linked AI summary

Baseline Shape Decides the Verdict: A Controlled Re-Examination of Ternary Language Models at 60K Parameters

Gautam Veldanda

arXiv:2609.29397v1cs.CL

TL;DR

Evidence on ternary language models below 1M parameters is limited, despite their appeal for microcontroller-scale budgets. The paper reruns a routed-model comparison under one recipe with three seeds per cell and examines baseline shape, architecture, ternary penalties, and FP-init training. It finds baseline shape can account for part of the short-budget advantage, while larger-budget comparisons and ternary-penalty estimates have important qualifications.

  • Problem

    Below 10M parameters, evidence on ternary language models is largely anecdotal, with prominent routed-model comparisons based on two single-seed runs.

    Source: §1 · p. 1

  • Method

    The authors rerun the comparison with one fixed recipe, three seeds per cell, token-matched full-precision anchors, and 98 byte-level runs.

    Source: §1 · p. 2

  • Results

    At 60K parameters, transformer shape shifts validation loss by 22.6%, more than replacing the architecture does, making the published result partly dependent on baseline shape.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A fixed baseline shape can flip the architecture verdict at 16M bytes, so conclusions from a single param-matched transformer shape are not robust to shape choice.

    Source: §1 · p. 2

    The transformers retain full-precision positional embeddings comprising 11–22% of parameters, so architecture-dependent ternary penalties cannot be attributed to architecture alone.

    Source: §7 · p. 9

Abstract

from arXiv · show

Ternary (1.58-bit) weights are attractive for microcontroller-class language models, but the sub-1M-parameter regime rests mainly on isolated, single-seed comparisons. One prominent example reports that a routed ternary block (convolution, diagonal SSM and sparse attention mixed by a per-token router) beats a parameter-matched full-precision transformer by 22% at 60K parameters, attributing this to inductive bias. We re-run it under one fixed recipe, three seeds per cell, 98 byte-level runs on one laptop. (i) Baseline shape dominates: at a 16M-byte budget, param-matched transformers span 22.6% in validation loss purely by depth/width choice - far more than any architecture effect we measure there - and the best-shaped transformer ties the routed model, so the published margin is at least partly a baseline-shape effect; the ordering of shapes reverses with budget, so no single fixed shape can be trusted. (ii) At 130M bytes the routed model does win, by 22.2-24.0% over the three transformer shapes we evaluate there - but a plain gated diagonal-SSM block beats it by a further 9.1%, and the routed model's own router puts most of its weight on its recurrent pathway, so the gain does not require routing. (iii) The ternary penalty differs by architecture at the larger budget (+5.3% best transformer vs. +19.5% routed, +28.1% gated SSM), but we cannot attribute that to architecture alone: our transformers keep learned positional embeddings in full precision, 11-22% of their parameters, so they are less quantized than the models they are compared with. (iv) A 90/10 full-precision-then-ternary schedule beats all-ternary training, but only at a stage-2 learning rate about 10x the pretraining peak; at a conventional fine-tuning rate it looks 15.3% worse, reversing the conclusion. The from-scratch baseline was not itself learning-rate tuned, which bounds (iii) and (iv). Code and run logs released.

1 Introduction

The paper re-examines whether ternary models have an architectural advantage at microcontroller scale, where prior evidence below 10M parameters is limited and often based on single-seed comparisons. Its controlled study finds that baseline shape can change the verdict, alongside budget-dependent results and training-related caveats.

1 Introduction

  • Below 10M parameters, the evidence is largely anecdotal, including a routed ternary model’s two single-seed comparisons against parameter-matched FP32 transformers.The reported routed-model advantage is 22% perplexity at 60K parameters, while it loses by ∼11% at 944K.Source: §1 · p. 1
  • At 60K parameters, transformer depth/width choice shifts validation loss by 22.6%, more than replacing the architecture does.The published result therefore depends partly on which single transformer shape is used as the baseline.Source: §1 · p. 2
  • The audit finds the published 60K and 944K comparisons differ in training tokens, corpus, schedule, optimizer settings, and baseline FFN width.The 944K comparison also changes precision and architecture at once.Source: §1 · p. 2
  • The controlled study uses 98 runs, three seeds per cell, one fixed recipe, token-matched full-precision anchors, and a 2σ seed-noise criterion.It evaluates baseline-shape sensitivity, three architecture families, ternary penalties across budgets, and a tuned FP-init recipe.Source: §1 · p. 2
  • A fixed transformer baseline shape can flip the architecture verdict at 16M bytes, while an untuned stage-2 learning rate can flip the FP-init verdict.The study reports 98 runs in ∼40 device-hours on one MacBook Air, with code and logs released.Source: §1 · p. 2

2 Related Work

Prior low-bit studies largely examine models above roughly 100M parameters, while small-model evidence has focused on transformers or isolated comparisons. This paper situates its study at MCU scale, where depth/width choices and training budget may alter architecture rankings.

Low-bit training and scaling

  • Spectra trains ternary transformers from 99M to 3.9B parameters, while ParetoQ evaluates 1–4-bit QAT from 125M to 8B; these studies therefore begin at roughly 100M parameters.ParetoQ reports that spending ∼90% of the budget in full precision before switching to QAT outperforms low-bit training from scratch, with ≤2-bit models requiring more tokens to saturate.Source: §2 · p. 2

Sub-100M quantization

  • Thomassen’s 720-run factorial studies INT8/INT6/INT4 QAT on transformer models from 3M–350M parameters, whereas this paper studies ternary weights and three architecture families down to 60K.TernaryLM trains a 132M ternary transformer with full-precision embeddings and head.Source: §2 · p. 2

Non-transformer low-bit models

  • Atome-LM is the only publicly released routed ternary hybrid at MCU scale the authors are aware of.Other cited non-transformer low-bit work applies grouped QAT to a pretrained 1.3B Mamba-2 model.Source: §2 · p. 2

Depth, width and the shape of a small model

  • At a fixed parameter budget, the depth/width optimum can move with compute or data budget, yet small-model comparisons often report only one parameter-matched baseline.The paper’s shape sweep finds the best shape at one budget is the worst at another.Source: §2 · p. 2

Inductive bias and scale

  • Prior work reports that the best-performing architecture can fluctuate at different scales, while this study varies training budget and precision as well as architecture.The paper asks whether the fluctuation at MCU scale is a size effect at all.Source: §2 · p. 3

3 Background: the Atome MCU block

The Atome MCU block combines convolution, diagonal state-space recurrence, and sparse causal attention through a per-token router, then adds a residual connection.

3 Background: the Atome MCU block

  • Each block applies LayerNorm, runs three parallel pathways, mixes them with a per-token softmax router, and adds a residual connection.The pathways are depthwise causal convolution, diagonal SSM recurrence, and single-head top-k causal attention; there is no feed-forward layer.Source: §3 · p. 3
  • Tokens are raw bytes with vocabulary 256, and the block uses no positional embedding.Channel mixing occurs only in the attention value projection and router.Source: §3 · p. 3
  • The block ternarizes its linear, convolution, embedding, unembedding, and router weights using per-tensor absmean scaling with an STE.The SSM’s per-channel vectors remain FP32.Source: §3 · p. 3

4 Auditing the published comparison

The audit reconstructs the conditions behind the two published Atome-LM comparisons and finds that multiple training and baseline differences complicate their comparison. In particular, the comparisons do not isolate architecture from precision, training regime, or baseline shape.

4 Auditing the published comparison

  • The two published comparisons differ in training budget and recipe, and the 944K comparison changes precision and architecture at once.The 60K and 944K runs differ by ∼650× in training tokens, as well as corpus, schedule, optimizer settings, and baseline FFN width.Source: §4 · p. 3
  • The released quantizer registry has no identity option, so no full-precision routed model was trained to isolate precision from architecture.The audit also notes that the 60K transformer baseline uses FFN width 1 × d, while the 944K baseline uses 4 × d.Source: §4 · p. 3
  • The 60K runs make roughly six passes over a 0.5 MB corpus, while the study’s budgets stay under 0.07 epochs.The authors do not re-run the published setup and cannot separate memorization from architecture in that comparison.Source: §4 · p. 3

5 Experimental design

The study uses a fixed byte-level recipe and three-seed comparisons to examine model shape, architecture, precision, and initialization across two training budgets. It specifies the data, evaluation rule, and model configurations used for those comparisons.

Hypotheses

  • The design specifies hypotheses about budget-dependent routed-model advantage, FP initialization, and architecture-dependent ternary penalties, while leaving the quantization-granularity hypothesis untested.The budget-advantage hypothesis is stated to apply independently of precision.Source: §5 · p. 4

Initialization

  • The FP-init arm trains in FP32 for 117M bytes, then ternary for 13M bytes, testing a 90/10 budget split at two stage-2 learning rates.Its transformer uses the 1 × 52 shape rather than the prespecified 3 × 32 shape, with a matching from-scratch reference.Source: §5 · p. 4

Run design

  • 98 distinct trainings compare routed and transformer models across 16M- and 130M-byte budgets, with a parameter-matched transformer shape sweep and a gated diagonal-SSM arm.The study reports 100 run directories because two tuned FP-init seed-0 runs reuse sweep runs.Source: Table 2 · p. 4, §5 · p. 4

Recipe and data

  • All runs use the published 944K recipe, and TinyStories V2 raw bytes are split into 256-byte chunks with 10% held out.The budgets correspond to 0.008 and 0.064 epochs, respectively; held-out chunks come from the same corpus and near-duplicate stories may cross the split.Source: §5 · p. 4

Model configurations

  • Every model ternarizes its token embedding, unembedding, and weight matrices, while per-channel SSM scalars and LayerNorms remain full precision.Transformers also retain learned positional embeddings in full precision; the gated SSM is 5.6% larger than the routed model, a caveat for their comparison.Source: Table 3 · p. 5, §5 · p. 5

Evaluation

  • Validation loss in nats/byte on the last checkpoint is the primary metric, and a gap counts as real only when it exceeds 2σ.Each ternary run is compared with a full-precision anchor trained on the same total bytes; comparisons with fewer than two seeds per group are undetermined.Source: §5 · p. 5

6 Results

At 130M bytes, the routed model beats the evaluated transformer shapes, but a gated diagonal-SSM does better still; at 16M bytes, transformer shape choice alone produces a larger loss span than the measured architecture effects. The ternary penalty, router behavior, and FP-then-ternary result all depend on important qualifications, including unequal quantization and learning-rate choice.

6.1 Baseline shape decides the verdict

  • At 16M bytes, transformer shapes span 22.6% in full-precision validation loss, while the routed model is only 0.95% better than the best shape in full precision and 4.0% better in ternary, both within noise.The best shape is 1 × 52; the worst is 6 × 24, and shape varies jointly with depth, width, FFN ratio, and positional-embedding share.Source: §6.1 · pp. 5–6, Figure 1 · p. 6

6.2 At a larger budget the gain is real, but it does not require routing

  • At 130M bytes, the routed model beats the three evaluated transformer shapes by 22.2–24.0% in full precision and 13.7–14.4% in ternary, while the gated diagonal-SSM beats it by 9.1% in full precision.The gated SSM matches the routed model in ternary; the comparison does not isolate recurrence as the causal ingredient because the blocks also differ in feed-forward layers and channel mixing.Source: §6.2 · pp. 6–7

6.3 The ternary penalty differs by architecture, with caveats we cannot yet remove

  • At 130M bytes, the ternary penalty is +5.3% for the best transformer shape, +19.5% for the routed model, and +28.1% for the gated SSM.The comparison is confounded because transformer positional embeddings remain full precision while recurrent models quantize more of their parameters; the from-scratch ternary baseline is also untuned.Source: §6.3 · p. 7

6.4 A 90/10 FP-then-ternary schedule helps, at a learning rate 10× the pretraining peak

  • In a token-matched 90% full-precision/10% ternary allocation, stage-2 rate 1×10^-4 looks clearly worse than ternary from scratch, while 3×10^-3 (10× the 3×10^-4 pretraining peak) wins.The five-point stage-2 sweep had not saturated at its upper end, so 3×10^-3 is a lower bound on the useful rate, not an estimated optimum; the comparison tests this 90/10 allocation, not quantization-aware fine-tuning in general.Source: §6.4 · p. 7

6.5 What the router actually does

  • At the end of a 130M-byte ternary run, the router assigns the state pathway mean per-token weights of 0.49–0.73 across layers, more than either convolution or sparse attention.Entropy of 0.67–0.87 nats indicates a non-uniform mixture rather than collapse to one pathway or uniform routing.Source: §6.5 · p. 8

6.6 On the original H-budget hypothesis

  • Against the shape used in the original work, the routed model’s advantage grows from 13.3% at 16M bytes to 22.2% at 130M bytes, contrary to the tested H-budget hypothesis.The two published points differed by about 650× in training bytes.Source: §6.6 · p. 8

7 Limitations

The study covers one parameter count, one byte-level corpus, and two byte budgets, limiting how broadly its architecture and ternary-penalty findings can be applied. Unequal quantization, untuned ternary training, and laptop-only timing further constrain interpretation.

7 Limitations

  • The study tests only 60K parameters, so it cannot establish whether architecture rankings or ternary penalties change at 1M or 10M parameters.A comparable 944K grid would require roughly two orders of magnitude more compute than the laptop allows.Source: §7 · p. 8
  • The results come from byte-level TinyStories V2, whose locally repetitive structure may favor convolutional and recurrent pathways over attention.Whether the findings persist on code, multilingual text, or subword-tokenized data remains open.Source: §7 · pp. 8–9
  • The study uses only 16M- and 130M-byte budgets, and the shortest budget spans just 245 optimizer steps.The budgets also represent low bytes per parameter: approximately 260 and 2,140, respectively.Source: §7 · p. 9
  • Transformer shapes share one recipe without per-shape learning-rate tuning; the gated SSM is deliberately simple, quality is measured only by loss, and timing is from a laptop rather than an MCU.MCU deployment cost is left for follow-up work.Source: §7 · p. 9
  • Transformers retain 11–22% of parameters in full-precision positional embeddings, versus 1.6–2.3% for recurrent models, making their ternary arms less quantized.The measured architecture-dependent ternary penalty is therefore an upper bound on the architectural component pending a zero-parameter position-scheme replication.Source: §7 · p. 9
  • From-scratch ternary runs reuse the full-precision peak learning rate, while the FP-init sweep had not saturated at 3×10−3, potentially inflating ternary penalties and leaving FP-init results as a lower bound.The paper identifies per-architecture ternary learning-rate sweeps as a way to tighten the penalty estimates.Source: §7 · p. 9

8 Conclusion

The controlled re-run finds that baseline shape partly explains the 60K routed-model advantage, while at longer budgets a gated recurrence does at least as well without routing. The architecture-dependent ternary penalty and the full-precision-then-ternary schedule results remain qualified by training and quantization limitations, and scaling beyond the tested setting remains unsettled.

8 Conclusion

  • At 60K parameters, the routed-model advantage is at least partly a baseline-shape artifact; at longer budgets, a gated recurrence does at least as well without routing.The comparison does not isolate recurrence itself as the cause.Source: §8 · p. 9
  • The ternary penalty differs sharply by architecture at the larger budget, but unequal quantization and an untuned from-scratch baseline prevent treating this as an established architectural law.Transformers retain 11–22% of their parameters in full precision, and loss level also covaries with architecture.Source: §8 · p. 9
  • A 90/10 full-precision-then-ternary schedule beats all-ternary training only at a learning rate far above the usual fine-tuning range.The from-scratch baseline was not learning-rate tuned, so the win's size and sign remain provisional.Source: §8 · p. 9
  • The study does not settle whether architecture ordering or architecture-dependent ternary penalties hold, shrink, or invert between 60K and 1M parameters.A comparable grid at 944K parameters was beyond the laptop budget; the tested scope also does not extend beyond TinyStories.Source: §8 · p. 9

A Exact faster kernels

The faster-kernel appendix describes a chunked scan for diagonal SSMs and gathered top-k attention. The SSM substitution matches the original computation closely while reducing measured runtime, and the attention path avoids building a full score matrix.

A Exact faster kernels

  • The diagonal SSM is trained as a depthwise convolution with a length-L kernel, using chunks of T=32 and carrying state across chunks.Within each chunk, the recurrence is applied as a lower-triangular T × T matrix per channel.Source: A Exact faster kernels · p. 10
  • The chunked SSM scan matches the original forward pass and gradients within 2 × 10^-6 relative error and cuts CPU runtime from 155 ms to 8 ms.The CPU measurement is 19× lower cost at C=L=256 and batch 16; the MPS timing was taken while another training job shared the GPU.Source: A Exact faster kernels · p. 10
  • Gathered top-k attention avoids scattering scores into an L × L matrix because entries outside the selected top-k are exactly zero after softmax.The implementation agrees within 1.2 × 10^-6, and the toy pipeline preserves the reported ternary–FP relative gaps.Source: A Exact faster kernels · p. 11

Related papers

Loading 2609.29397v1…