Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale

2. Learnability: can a shared network learn useful language computations from that interface?

• The paper separates whether an injective code distinguishes tokens from whether a shared network can learn useful computations or whether adapting each token vector improves the model.An injective code alone answers only the coding question.Source: §2 · p. 1
• The fixed interface keeps token coordinates unchanged while the remaining trainable system adapts, creating a restriction for studying how lexical and contextual representations are constructed.This removes one source of representational change without making the resulting representations automatically interpretable.Source: §2 · p. 1
• For 49,152 tokens, the binary identity uses K = 16 coordinates, repeated to Transformer width without a trainable projection; a second model applies an invertible GF(2) recoding.The experiment treats binary codes as a transparent, collision-free construction, not as uniquely suitable for language.Source: §2 · p. 2
• Fixed-code models retain substantial capability but underperform the learned-input control on important tasks, so the learned table is useful but not required for the demonstrated capabilities.The evaluation spans standard multiple-choice tasks, last-word prediction, and corpus likelihood.Source: §2 · p. 2
• The study contributes a backbone-matched comparison, measures retained capabilities and quality cost, tests one noncanonical recoding, and frames the fixed interface as a setting for representation-learning studies.The recoding establishes viability for one assignment, not invariance across assignments.Source: §2 · p. 2

2. Related Work

• Prior work explores tied, factorized, shared, generated, and frozen input alternatives; this study characterizes a restrictive fixed-binary interface at scale.The interface uses fixed minimal binary identities, a fixed lift, and no learned input generator before the Transformer backbone.Source: §2 · pp. 2–3

3.1. Learned input and output parameterizations

• Each token ID selects an independently adjustable learned input row, while every model retains an untied, trainable output projection.Eliminating trainable input rows therefore does not remove vocabulary-specific trainable parameters from the complete model.Source: §3.1 · p. 3

3.2. Canonical binary identities

• For 49,152 tokens, the fixed-length binary identity uses 16 bits; those bits are not assumed to encode linguistic attributes or semantic neighborhoods.“Minimal” refers only to the width of an injective fixed-length binary code.Source: §3.2 · p. 3

3.3. Deterministic lift

• The 16-coordinate code is repeated to width 2048 through a full-column-rank lift with no trainable parameters.No learned input projection, codeword vectors, or auxiliary embedding generator precedes the trainable backbone.Source: §3.3 · p. 3

3.4. Fixed invertible recoding

• The fixed invertible recoding preserves token identity but not Hamming distance, input norms, or optimization behavior; only one recoding is evaluated.The vocabulary occupies only a subset of the binary hypercube, so full-hypercube balance properties do not automatically apply.Source: §3.4 · p. 4

3.5. Implementation and scope

• The fixed interfaces store codebook values, and the evaluated implementations use a compact-buffer lookup rather than a literally lookup-free mechanism.The tokenizer is corpus-derived, so fixing token coordinates does not remove every possible source of corpus information from the input pipeline.Source: §3.5 · p. 4

4.1. Models and controlled comparison

• The three models share a decoder backbone, output head, preprocessing, sampling, and optimization recipe; only input parameterization changes, without reallocating removed parameters.This is a backbone-matched component intervention, not an equal-parameter, equal-FLOPs, or equal-wall-clock comparison.Source: §4.1 · p. 4

4.2. Data and sampling

• Training uses FineWeb-Edu-dedup without Cosmopedia; at context length 2048, documents with at least 2049 tokens are eligible, and documents then valid window starts are sampled uniformly within the resident shard.Source: §4.2 · p. 5

4.3. Optimization and replication

• Each interface targets 100B prediction tokens, but only one run is trained per interface and interrupted runs may not reproduce sample order.The recipe uses AdamW with warmup and cosine decay; realized budgets are determined by checkpoint counters rather than launch arguments.Source: §4.3 · p. 5

4.4. Evaluation

• Evaluation covers zero-shot multiple-choice tasks, five-shot MMLU, LAMBADA last-word prediction, and WikiText likelihood, with task-specific metric normalization.The compact main table is not an aggregate capability score, and reported standard errors do not estimate training-seed variability.Source: §4.4 · p. 5
• The selected evaluations show that fixed-code models retain substantial capability but do not match the learned-input control across the suite.Table 2 presents selected accuracy metrics and standard errors; each column represents one training run, and complete results appear in Appendix A.Source: Table 2 · p. 6

4.5. External references

• SmolLM2 models provide external quality references, not matched controls for sample efficiency or the input-interface effect.They differ in size, architecture, data, schedule, and compute; their larger token budgets do not support an efficiency claim for fixed codes.Source: §4.5 · pp. 5–6

5.1. Fixed identities support substantial capability

• The canonical fixed-code model achieves 52.40% HellaSwag normalized accuracy, 70.51% PIQA accuracy, and 42.75% LAMBADA accuracy, with useful WikiText corpus likelihood across evaluation formats.HellaSwag and PIQA performance is well above uniform-choice reference levels, while LAMBADA and WikiText demonstrate nontrivial next-token prediction.Source: §5.1 · p. 6

5.2. A noncanonical assignment remains viable

• The recoded model performs in the same broad range as canonical codes, but only one matrix is tested and metric ordering varies across the two assignments.Canonical codes do better on HellaSwag and WikiText, while the ordering reverses on some other metrics; this does not establish invariance to recoding.Source: §5.2 · p. 6

5.3. Near-chance tasks delimit the evidence

• Approximately 20% CommonsenseQA accuracy and 25–26% MMLU scores remain near uniform-choice references, delimiting the demonstrated capabilities to informative tasks.The results do not establish broad knowledge or general reasoning.Source: §5.3 · p. 6

5.4. External references calibrate absolute quality

• Fixed-code models exceed SmolLM2-135M on several metrics but trail SmolLM2-360M on others; SmolLM2-1.7B is substantially stronger on the reported suite.Canonical-code HellaSwag normalized accuracy is 52.40%, between SmolLM2-135M at 43.02% and SmolLM2-360M at 56.28%.Source: §5.4 · p. 6

6.1. Necessity and utility are different claims

• Fixed-code models retain substantial capability despite an observed quality cost, showing feasibility at the tested scale and budget—not universal optimality or equivalence across training conditions.The learned input table benefits several evaluations, but its usefulness does not establish that it is required for the behavior under investigation.Source: §6.1 · p. 7

6.2. Removing a table does not remove an embedding function

• Shared trainable transformations can construct lexical and contextual representations from fixed coordinates, but they are not the same parameterization as independently adjustable lookup-table rows.The raw-input rank restriction does not extend to later hidden states, where RMSNorm, nonlinearities, attention, and residual connections operate.Source: §6.2 · p. 7

6.3. An immutable input controls a boundary, not an entire system

• Immutable token inputs provide an experimental boundary, but do not make hidden representations unique, align independently trained networks, or localize capabilities to a particular layer.The models are trained from scratch; the experiment does not show that semantic information was transferred from an existing table, and the output projection remains trainable and vocabulary-specific.Source: §6.3 · p. 7

6.4. Binary minimality is an experimental choice

• The 16-bit binary interface makes identity preservation and the absence of trainable input coordinates explicit, but its dimension follows from this vocabulary rather than prescribing a universal language-model dimension.Other fixed encodings would need separate checks of collision behavior, precision, scale, and learnability; these results do not establish that they would perform equally well.Source: §6.4 · pp. 7–8

Representation formation without a moving input

• Fixed token identities let researchers track when lexical and contextual distinctions become accessible across training and depth.The passage recommends pairing representational measurements with causal interventions, such as layer ablations, activation substitutions, and matched-context tests.Source: §7 · p. 8

Separating code geometry from trainability

• Comparing learned low-dimensional coordinates, fixed random codes, norm-matched controls, and multiple recodings could help separate effects of fixedness, dimension, scale, and code assignment.Such comparisons would investigate why the interface works, beyond showing that it can work.Source: §7 · p. 8

Stable boundaries for modular systems

• A reproducible token-identity interface could remain unchanged while contextual computation, memories, or executable components are separately replaced or studied.The proposed operational benefit is a stable input specification, not a demonstrated production advantage.Source: §7 · p. 8

• One training run per interface means reported differences describe these checkpoints, not variability across initializations or realized data streams.Task-level standard errors cannot substitute for independent training runs.Source: §8 · p. 8
• Input trainability, parameter count, geometry, and scale change together, so the quality gap cannot be attributed to a single mechanism.The fixed zero/one vectors have different initial norms from the learned vectors, and the shared recipe was not independently optimized for each interface.Source: §8 · p. 8
• Generalization is limited to one tokenizer family, vocabulary size, context length, data configuration, and main model scale; multilingual, long-context, instruction-tuned, and substantially different-budget settings remain unestablished.Near-chance MMLU and CommonsenseQA scores particularly limit claims about broad reasoning and knowledge.Source: §8 · p. 8
• No throughput, energy, or latency advantage was measured, and removing input parameters does not by itself establish a runtime improvement.Runtime depends on buffer access, tiling, activations, and kernel behavior; with tied input and output weights, the matrix may remain in storage.Source: §8 · p. 8
• The intervention concerns input parameterization, not elimination of all learned lexical storage, because the output head retains a full vocabulary-sized trainable matrix.Source: §8 · p. 8

• Fixed minimal token identities support substantial capability at 1.7B-class scale without independently trainable token-specific input vectors, while the learned-input control performs better on several informative evaluations.Source: §9 · p. 9
• The result separates input-table utility from necessity: the table is useful in the tested setting, but not required for the capabilities observed.Source: §9 · p. 9
• Immutable input identity creates a controlled setting for studying representation learning in the remaining trainable system, without establishing capability mechanisms or locations.Source: §9 · p. 9

Reproducibility Statement

• The released materials include training implementations, configurations, checkpoint-loading utilities, and dataset-preparation and training source code.The artifacts cover Learned, Binary16, and GF2 models.Source: Reproducibility Statement · p. 9
• The verification utility checks codebooks, the GF2 transformation, loading, logits, and short generation, but does not prove full training–inference equivalence.The checks are implementation checks, not benchmark evaluations.Source: Reproducibility Statement · p. 9

Ethics Statement

• The fixed input interface does not establish factual reliability, interpretable reasoning, or control over generated content.The checkpoints are base research models, not validated instruction-following assistants.Source: Ethics Statement · p. 9

A. Complete Evaluation Results

• Table 3 reports the complete evaluations; accuracy errors are in percentage points, while perplexities and bits per byte are not percentages.Five-shot MMLU includes context-truncated candidate requests, and reported errors are evaluation standard errors rather than training-run uncertainty.Source: Table 3 · p. 11, A. Complete Evaluation Results · p. 11

B. Architecture and Training Details

• Fixed-code and learned-input models retain the same backbone construction, but their initial input scales are unmatched and raw inputs also enter the residual stream.RMSNorm affects attention input but does not make input scaling irrelevant.Source: B. Architecture and Training Details · p. 12
• Uniform document sampling within resident shards, window sampling, and replacement mean training is not one pass through all corpus tokens.Documents shorter than 2049 tokens are excluded from training-window sampling.Source: B. Architecture and Training Details · p. 13

C. Evaluation Audit and Interpretation

• Matching sample and prompt multisets supports consistent evaluated text but does not independently validate likelihood alignment or every adapter path.About 2.68% of five-shot MMLU candidate requests were marked truncated, so the result is not an unrestricted full-prompt evaluation.Source: C. Evaluation Audit and Interpretation · p. 13

D. Elementary Properties and Boundary Cases

• An injective binary code for V symbols requires K ≥ ⌈log2 V⌉, and the canonical construction attains this bound as a coding fact.The bound does not establish language-model learnability.Source: D. Elementary Properties and Boundary… · p. 13
• Both fixed interfaces are collision-free on the vocabulary, but their shared input coordinates do not determine hidden-state bases or guarantee modular compatibility.The raw input rank restriction applies to input representations, not every later activation matrix.Source: D. Elementary Properties and Boundary… · pp. 13–14
• An all-zero sequence of zero-code tokens yields zero logits and a uniform next-token distribution in the supplied bias-free architecture, assuming finite parameters.A zero-code token after nonzero context can still receive information through causal attention.Source: D. Elementary Properties and Boundary… · p. 13

E. Artifacts

• The release provides dataset-preparation and training scripts, including model source code, in the supplementary material.The model checkpoints and loading utilities are listed in Appendix E.Source: E. Artifacts · p. 14