PaperlayerView exact revision on arXiv

Source-linked AI summary

Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models

Sai Kiran Bathula

arXiv:2610.22428v1cs.CL

TL;DR

Small models devote a large share of their parameters and compressed-file size to learned vocabulary tables, raising the question of what those tables buy. The paper replaces the table with frozen codes derived from a pretrained embedding and compares models trained with the same data. The frozen-table model is better or tied on three of four benchmarks, while the learned table performs better on ARC-Easy.

  • Problem

    In Gemma 3 270M, the embedding table contains 168M of 268M parameters, and quantisation leaves the table in 16-bit while compressing the body.

    Source: §1 · p. 1

  • Method

    The paper derives fixed 512-bit token codes from a pretrained embedding and trains the same model body with a learned table or frozen codes.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    The frozen-table model is 1.6 points better than the learned-table model on LAMBADA, tied on HellaSwag and PIQA, and 3.4 points worse on ARC-Easy.

    Source: §4 · p. 3

  • Takeaways & Limitations

    On tokens seen 10 to 99 times in training, the frozen table wins by 2.3 nats for B and 1.7 nats for C.

    Source: §4 · p. 4

    The CPU comparison covers one laptop and two different code bases, and reports generation speed only because the kernel’s prefill is one token at a time.

    Source: §6 · p. 7

Abstract

from arXiv · show

In a small language model the vocabulary is the single largest part. In Gemma 3 270M the embedding table holds 168M of 268M parameters, and because quantisation methods leave the embedding in 16-bit, it is also most of the file once the rest of the model is compressed. We ask what that table actually buys. We replace it with one frozen table of 512-bit codes, 16 MB, with no trainable parameters, derived once from a pretrained embedding by principal-component projection and sign binarisation, and we train the same Gemma-shaped body from scratch on the same 2.05B tokens with and without it. The learned table buys 0.20 nats of validation loss for 2.6 times the trainable parameters, but on LAMBADA, HellaSwag and PIQA the frozen-table model is better or tied; only ARC-Easy favours the learned table. With a ternary BitNet b1.58 body the whole model is a 46 MB file, still ties the float model on LAMBADA and PIQA, and on a laptop CPU generates 615 to 740 tokens per second in 62 MB of memory, five to six times the speed of the float model's 8-bit GGUF in llama.cpp at a seventh of the memory. The frozen table wins by 1.7 to 2.3 nats on rare words, accepts 10,000 new words after training with a 3% cost where the learned table collapses by 43%, and all of it is released with the weights, the data and every evaluation answer.

1 Why the vocabulary

For small language models, the learned vocabulary table can dominate parameter count and remain most of the file after body quantisation. Earlier frozen-table models motivated a controlled comparison that changes only the vocabulary.

1 Why the vocabulary

  • 168M of Gemma 3 270M’s 268M parameters belong to its embedding table, while its feed-forward layers contain 71M.The table is usually tied to the output head, and quantisation leaves it in 16-bit while compressing the body.Source: §1 · p. 1, Table 1 · p. 2
  • After body compression, the 16-bit vocabulary can dominate storage; a ternary-body Gemma 3 270M would have about 20 MB of body and 335 MB of vocabulary.The passage also notes BitNet reports memory as “0.4 GB (non-embedding)” [3].Source: §1 · p. 1
  • Earlier frozen-table models showed feasibility but changed multiple factors at once, so this study holds body, data, steps, and seed constant while changing only the vocabulary.SHADOW-250M and SHADOW-50M used fixed codes, but neither was compared against a learned table under controlled conditions.Source: §1 · p. 2

2 The frozen table

The proposed frozen vocabulary encodes pretrained embeddings as compact binary codes and uses small trained maps for input and output. The codes retain measured similarity information and can also encode new token vectors.

2 The frozen table

  • The codes come from centering Gemma 3 270M’s 262,144-by-640 embedding, projecting onto its top 512 principal directions, and keeping each projection’s sign.The resulting binary code is fixed rather than updated during training.Source: §2 · p. 2
  • The frozen table stores 512-bit codes for 262,144 tokens in 16.8 MB, with two trained interface maps totaling 0.65M parameters and a fixed unigram-prior seal costing 1 MB.The output scores each code by dot product divided by sqrt(512); Arm C trains the prior, adding 262,144 trainable numbers.Source: §2 · p. 2
  • The 512 directions retain 92.3% of embedding variance, and code similarity correlates with WordSim-353 human ratings at 0.617 versus 0.614 for the float embedding.67% of each token’s ten nearest float neighbours also appear among its ten nearest code neighbours.Source: §2 · p. 2
  • Because codes are computed from embedding-space vectors, a new word can use the code of the mean of its pieces’ rows or of any chosen vector.A learned table has no corresponding rule, and an added row is random until trained.Source: §2 · p. 2

3 Experiment

The experiment trains three Gemma-shaped models from scratch, comparing a learned table, a frozen table with a float body, and a frozen table with a ternary body. Arms share data batches and seed, while the ternary arm also uses a different published training recipe.

3 Experiment

  • All arms train from scratch with the Gemma 3 270M architecture, including 18 layers and width 640.The configuration also specifies gated GELU FFNs of 2,048 and local sliding-window attention layers.Source: §3 · pp. 2–3
  • Arm A uses a learned tied 262,144-by-640 table with a bfloat16-autocast body; Arm B uses the frozen table and fixed seal with the same float body.Arm C uses the frozen table with BitNet b1.58 BitLinear body layers and a trained seal; trainable parameter counts are 268.1M, 101.0M, and 101.2M, respectively.Source: §3 · p. 3
  • Each arm trains on the same 2.05B FineWeb-Edu tokens and batches for 7,820 steps, with a shared seed; Arm C instead uses BitNet’s published schedule without tuning.Arms A and B use AdamW with a 6e-4 peak learning rate; C uses a separate two-stage recipe.Source: §3 · p. 3
  • Each arm trained on one RTX 5090 for 14 to 16 hours, and the study including evaluation cost under 75 dollars.The reported rental cost was about 18 US dollars per arm.Source: §3 · p. 3
  • Evaluation uses 512 held-out 1,024-token sequences for validation loss and zero-shot full-test evaluation on LAMBADA, HellaSwag, ARC-Easy, and PIQA.Error bars are 95% bootstrap ranges over questions, with arm comparisons paired on the same questions.Source: §3 · p. 3

4 Results

The learned table improves validation loss, but its benchmark advantage is limited; the frozen-table ternary model remains close to the float model while using a much smaller file. The frozen table’s strongest loss advantage is on infrequent tokens.

What the learned table buys

  • The learned table buys 0.20 nats of validation loss; B is 1.6 points better than A on LAMBADA, tied on HellaSwag and PIQA, and 3.4 points worse on ARC-Easy.The paired-difference table defines “tied” as a 95% range that includes zero.Source: §4 · p. 3, Table 3 · p. 3

How the gap develops

  • B starts 0.23 nats below A at step 200, but A overtakes at step 600 and holds a 0.20-nat loss advantage from about step 3,000 to the end.Validation loss was checked every 200 steps over 2.05B training tokens.Source: §4 · pp. 3–4, Figure 1 · p. 5

Where the gap is

  • On tokens seen 10 to 99 times, the frozen table wins by 2.3 nats (B) and 1.7 nats (C); the learned table wins for tokens seen 100 to 100,000 times.The 161 most common tokens, 52% of all text, have a 0.035-nat gap; 52,782 vocabulary tokens never appear in training and 147,187 appear at least ten times.Source: §4 · p. 4, Table 4 · p. 4

The ternary model

  • C’s 46.2 MB file ties A on LAMBADA and PIQA, trails by 0.9 points on HellaSwag and 4.9 on ARC-Easy, and is within 0.8 to 2.0 points of B across benchmarks.C uses a 2-bit-packed ternary body; against B, it is also 0.08 nats worse in loss.Source: §4 · p. 4

On a laptop CPU

  • C generates 5.3 to 6.4 times faster than A’s Q8_0 model in 7.6 times less memory.The GGUF 8-bit model’s perplexity is 23.864 against 23.857 for the float model, far inside the error.Source: §4 · p. 4, Table 5 · p. 4
  • At any quantisation, A remains above 446 MB because its vocabulary table stays 16-bit; C’s speed is limited by streaming its 25 MB body and 17 MB table through memory.The reported laptop memory bandwidth is 38 GB/s, giving an estimated ceiling of about 860 tokens per second.Source: §4 · p. 5

5 Adding to the vocabulary after training

The paper tests whether frozen-table codes can accept vocabulary added after training. They accept 10,000 new words, but the tested additions cost quality; results also vary across languages and do not improve the control-token or emoji tests.

5 Adding to the vocabulary after training

  • The new-token test adds biomedical words split into two or more pieces by Gemma’s tokenizer, using piece-derived codes for B and C and mean-norm rows for A.The 10,000 words average 2.5 pieces; new-token priors for B and C use estimated training frequency selected on a validation split.Source: §5 · p. 6
  • 10,000 new words can be added without training, but the frozen-table model incurs a 3% cost while the learned table collapses.The added tokens replace 3.2% of tokens in the held-out articles; the frozen-table model also reads the text with 3% fewer tokens.Source: §5 · p. 6
  • Across 20 languages, B is better than A in Arabic, Bulgarian, Spanish, Russian, and Vietnamese, and worse in the other fifteen.For never-seen tokens, setting their prior to the median of seen-token priors makes B better than A in Thai, Arabic, Russian, French, and Greek; A remains ahead in Hindi, Japanese, and Chinese.Source: §5 · p. 6
  • At five shots, brand-new control tokens are placed correctly by every arm, while emoji prediction is at chance for all three models.At five shots, B scores 88%, C 75%, and A 73%; emoji prediction scores 2.6 to 5.7% against 5%.Source: §5 · p. 6

6 Limitations

The evaluation is limited by one training seed and a token budget that under-trains the learned table. The ternary-model comparison, the source embedding, vocabulary expansion, and CPU measurements also have specific scope boundaries.

6 Limitations

  • One seed per arm leaves run-to-run uncertainty unmeasured, and 2B tokens under-trains A’s table; its advantage may grow at 20B tokens, where the benchmark picture may also change.The authors expect the 20B-token loss gap to exceed 0.20, but did not measure that setting.Source: §6 · pp. 6–7
  • C changes three factors versus B—the ternary body, BitNet’s schedule, and trained prior—while the schedule’s contribution was not isolated.A separate B run found the trained prior 0.003 to 0.022 nats better at matched checks.Source: §6 · p. 7
  • The frozen codes derive from Gemma 3 270M’s embedding, inherit its knowledge boundaries, and carry Google’s Gemma terms; another initialization route remains untested.That untested route is to train a small model with a learned table, then freeze its codes for later models.Source: §6 · p. 7
  • Vocabulary expansion improves coverage only for words the tokenizer misses; adding tokens that the tokenizer already covers costs quality in every tested setting.The authors therefore bound the gain by what the tokenizer is missing.Source: §6 · p. 7
  • The CPU comparison covers one laptop and compares different code bases; absolute speeds depend on memory bandwidth, and only generation speed is compared.The C kernel’s prefill is one token at a time, and B has no kernel.Source: §6 · p. 7

7 Related work

Prior work covers frozen identity codes, embedding-table sharing or factorisation, and BitNet quantisation, but the paper distinguishes its pretrained-derived codes shared across input and output. It also positions its controlled comparison against the authors’ earlier releases.

7 Related work

  • The authors’ SHADOW-250M and SHADOW-50M introduced the frozen 512-bit table and CPU kernels; this paper adds the controlled comparison those releases lacked.Bochkov [8] [9] instead uses frozen identity codes on input and retains a trained float output head.Source: §7 · p. 7
  • Unlike Bochkov [8] [9], this work derives codes from a pretrained embedding, shares them across input and output, and removes the vocabulary matrix on both sides.The cited identity codes carry no meaning, and the output table remains.Source: §7 · p. 7
  • Tied embeddings [10], ALBERT factorisation [11], hash embeddings [12], and unigram output biases [13] retain trainable float components; arm C trains the prior while arm B freezes it.The cited output-bias method initializes a per-token bias to the log unigram distribution.Source: §7 · p. 7
  • BitNet [1] [2] [3] supplies the ternary body and schedule, while GPTQ [14] and AWQ [15] quantise the body but leave embeddings in 16-bit.The cited BitNet reports state model memory excluding the embedding.Source: §7 · p. 7

8 What we release

The release provides the materials needed to reproduce the reported results and add arms to the comparisons, under a non-commercial research licence and applicable Gemma terms.

8 What we release

  • The repository releases scripts, settings, table and unigram data, checksummed training and validation tokens, checkpoints, logs, evaluation scripts, and per-question results for recomputing numbers or adding arms.The release is licensed for non-commercial research; the table and B/C checkpoints containing Gemma-derived codes are also under Google’s Gemma terms.Source: §8 · p. 7

9 What comes next

The next paper proposes a frozen 512-bit code space shared by text, images, audio, and video, with one small body and no learned embedding. This passage gives a teaser rather than measurements.

9 What comes next

  • The proposed follow-up places text, images, audio, and video in one frozen 512-bit code space for a small body without learned embeddings.The passage says the measurements will come with the next paper.Source: §9 · p. 8

How to cite

The paper is a 2026 preprint by Sai Kiran Bathula; its DOI and code, data, and weights location are listed.

How to cite

  • Sai Kiran Bathula authored the 2026 preprint Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models.Source: How to cite · p. 8
  • The preprint's DOI is 10.5281/zenodo.23382634.Source: How to cite · p. 8
  • The code, data, and weights are available at huggingface.co/QLNI/embeddings-are-overpriced.Source: How to cite · p. 8

Related papers

Loading 2610.22428v1…