Bolmo: Byteifying the Next Generation of Language Models

1 Introduction

• Subword tokenization limits character-level understanding, especially for scientific data, while also introducing tokenization bias, English-centricity, and potentially suboptimal compute allocation.The cited examples of scientific data include code and biological sequences.Source: §1 · p. 3
• The authors hypothesize that byte-level models have lagged in adoption because most are trained from scratch, while keeping pace with subword-model advances requires extensive investment.Those advances include training data curation, architecture, and post-training.Source: §1 · p. 3
• Bolmo byteifies existing subword models with a specially designed LTLM architecture and two-stage training, enabling recovery and sometimes surpassing the source model’s performance.The architecture pools bytes into patches, processes them with a Transformer, then depools them into bytes.Source: §1 · p. 3
• Bolmo models average outperform comparable public byte-level LLMs, and Bolmo 7B achieves +16.5% absolute improvement in STEM tasks over BLT 7B.The comparison is against BLT 7B, which was trained from scratch.Source: §1 · p. 3
• Bolmo can be further sped up by increasing bytes per patch, and existing source-model ecosystem components can adapt it without additional training cost.The authors also report that byte-level models may improve computational efficiency, reduce energy and deployment costs, and support fine-grained textual understanding.Source: §1 · p. 4

2 Related Work

• Subword tokenization loses information about characters, can cause inference-time tokenization bias, imposes a fixed vocabulary, and ties compute allocation to tokens.The section notes that these constraints affect character-reasoning tasks, prompt endings, language efficiency, and compute allocation.Source: §2 · p. 4
• Direct byte-level models can address several subword-tokenization problems, but their longer sequences exacerbate compute allocation; latent-patch models pool bytes, process shortened sequences, and depool them.Dynamic tokenization is reported to improve the performance–efficiency Pareto front.Source: §2 · pp. 4–5
• UTF-8 byte-level models use a vocabulary of 256 bytes, though UTF-8’s Latin-script orientation may leave inefficiency for other languages.The section identifies alternative fine-grained units as a possible direction.Source: §2 · p. 4
• Tokenizer transfer changes a model’s tokenizer; byteification is a special case, and Bolmo adds an LTLM architecture and two-stage procedure to closely match subword-model performance.Earlier approaches used parameter initialization or boundary supervision with embedding matching.Source: §2 · p. 5

3.1 Architecture

• Byte embeddings add the longest matching source-subword suffix embedding, retaining sparse parameters for a better performance–efficiency tradeoff.The paper says similar performance can generally be reached by enlarging the local encoder instead.Source: §3.1 · p. 5
• Bolmo embeds each UTF-8 byte, contextualizes bytes locally, pools them into patches for the retained Transformer, then depools and decodes byte predictions.The local decoder and LM head produce next-byte predictions and boundary decisions.Source: Figure 1 · p. 6, §3.1 · p. 7
• Bolmo 1B has ∼10M fewer parameters than OLMo 2 1B (−0.7%), while Bolmo 7B has ∼330M more than Olmo 3 7B (+4.5%).The parameter counts remain similar by removing the output embedding matrix and adding local encoder and decoder parameters.Source: §3.1 · p. 7

3.1.1 Non-Causal Patch Boundary Prediction

• For prefill, Bolmo uses one byte of future context to set patch boundaries; during decoding, it predicts boundaries alongside bytes using fused boundary symbols.The authors report that one-byte lookahead largely matches subword tokenization behavior.Source: §3.1.1 · pp. 7–9

3.2 Byteifying Procedure

• Byteification initializes the global model from the subword checkpoint and trains local components in Stage 1 with the global model frozen, then trains the full model in Stage 2.Stage 1 is designed to recover the source model’s behavior before end-to-end training.Source: §3.2 · p. 9, §3.2.1 · p. 9

3.2.1 Stage 1: Subword-to-Byte Distillation

• The prefill boundary predictor achieves >99% accuracy when trained to reproduce subword-tokenizer boundaries.The predictor uses future context during prefill.Source: §3.2.1 · p. 10
• Stage 1 distills encoder representations after n frozen global layers and uses output-boundary prediction for exact patch-likelihood distillation of the decoder.Using n > 0 improves performance over n = 0 while remaining relatively inexpensive; the authors choose n = 4.Source: §3.2.1 · p. 10

3.2.2 Stage 2: End-to-End Training

• Stage 2 trains all parameters end-to-end with boundary and byte cross-entropy losses, allowing the global model to exploit byte-level information and optionally increase bytes per patch.Stage 1 assumes the local encoder and boundary predictor perfectly emulate the subword model, which is close but not exact in practice.Source: §3.2.2 · p. 11

Data

• Bolmo’s training mix contains ∼172B Dolma 3 tokens plus 75M CUTE-style tokens, and training runs for less than one epoch.The CUTE-style data is sampled to avoid overlap with the CUTE test set.Source: §4 · p. 11

Model

• The starting checkpoint is pretrained Olmo 3 7B after mid-training and long-context extension, with alternating mLSTM and feedforward layers in the local models.The encoder and decoder use feedforward layer sizes of 1 and 4, respectively.Source: §4 · p. 12

Training

• Stage 1 trains local components for 9.8B tokens with the global model frozen; Stage 2 trains the full model for 39.3B tokens.The reported totals correspond to approximately 43B and 173B bytes, respectively.Source: §4 · p. 12

Baseline

• The baseline continues training Olmo 3 7B on the Bolmo data for 39.3B tokens, matching the total gradient updates to global-model parameters.This comparison is intended to separate continued training effects from byteification.Source: §4 · p. 12

Ablations and Development

• Architecture decisions were developed mainly through OLMo 2 experiments, and the byteifying procedure was applied to Olmo 3 7B without adjustment.Larger training-configuration sweeps were conducted on OLMo 2 1B because an Olmo 3 1B version was unavailable.Source: §4 · p. 12

Evaluation

• The evaluation suites add CUTE and EXECUTE to assess character understanding in English and across other languages, respectively.The 7B suite omits GSM Symbolic and BigCodeBench because of their size; the 1B suite adds CUTE to the Base Easy Suite.Source: §4 · p. 12

5 Main Results

• Bolmo 7B leads public byte-level models in every category except GenQA (70.9 vs. 71.3) and comes close to source Olmo 3.Table 1 compares models on the Bolmo 7B evaluation suite; all models except Bolmo were trained from scratch.Source: §5 · p. 12, Table 1 · p. 13
• Bolmo 7B surpasses Olmo 3 on character understanding, including when Olmo 3 continues training on Bolmo’s data mix for the same token count.The authors suggest character understanding emerges sooner in byte-level models, while noting that scale also drives it.Source: §5 · pp. 12–13
• Bolmo 1B is competitive with byte-level peers, outperforming H-Net but trailing BLT 1B, whose reported parameter count excludes hash embeddings.Against OLMo2 1B, Bolmo 1B scores −3.2% on MMLU, +5.1% on Lambada, +3.3% on CoQA, and +32.5% on CUTE.Source: §5 · pp. 13–14, Table 2 · p. 14

5.1 Training at Higher Compression Factors

• Byte-level models take over the task-performance–efficiency frontier when the subword model’s softmax cost dominates, enabling higher efficiency with a smooth performance dropoff.For OLMo 2 1B, the subword-model threshold is between vocabulary sizes of 200k and 400k; the figure compares tokenizer transfer to SuperBPE with Bolmo’s adapted boundary prediction.Source: Figure 3 · p. 15, §5.1 · p. 16
• BPE merging outperforms entropy- and cross-entropy-based merging in this byteifying setting, though validating that pattern with training from scratch remains open.The methods merge subword boundaries toward a target average bytes-per-patch ratio, which the model does not necessarily attain.Source: §5.1 · pp. 14–16

5.2 Post-Training Byteified Models via Task Arithmetic

• Task Arithmetic raises Bolmo’s IFEval score from 31.1% to 67.4%, on par with the post-trained Olmo 3 checkpoint’s 66.9%.The transfer adds the difference between post-trained and base Olmo 3 Transformer-layer weights to the corresponding Bolmo layers, with no extra training cost.Source: §5.2 · p. 16, Figure 4 · p. 16
• Post-training transfer depends on embedding resettability: resetting embeddings generally preserves performance, especially for larger models, but does not always do so.The local encoder and decoder remain in the base model space while the Transformer layers are adapted.Source: §5.2 · pp. 16–17

6.1 Impact of Non-Causal Patch Boundaries

• Non-causal prediction of patch ends uses one future byte to simplify prediction while preserving coherent patches compatible with the global model.Causal prediction must trade off easy start-boundary prediction against harder end-boundary prediction by the shallow local encoder.Source: §6.1 · p. 17

6.2 Is Stage 1 Training Necessary?

• Stage 1 improves performance but is not necessary for a good final run; its key benefit is faster iteration, and the bits-per-byte gap narrows but remains in its favor.Figure 6 compares trajectories over matched remaining training tokens; the authors caution that learning-rate schedules limit extrapolation to larger token budgets.Source: §6.2 · pp. 17–18, Figure 6 · p. 18
• To approximately match compute, Stage 2-only training adds 6.5B tokens, or 17% more Stage 2 training, because Stage 1 requires fewer FLOPs.The comparison disregards Stage 1's lower memory requirements, which may slightly favor the Stage 2-only run.Source: §6.2 · p. 18

6.3 Selecting the Right Local Model Architecture for Fast Inference

• FLOPs and FLOPs/byte can mislead architecture choices because their relationship with wall-clock inference speed is inconsistent and FLOP counts depend on counting conventions.The authors recommend using wall-clock inference measurements to guide choices.Source: §6.3 · pp. 18–19
• At ∼6.6 bytes per patch, Bolmo 7B overtakes Olmo 3 7B; at matched compression, decoding is ∼125 bytes/s versus ∼150 bytes/s.Bolmo takes ∼1s to prefill 72K bytes, compared with ∼0.8s for the subword model on the corresponding tokens.Source: §6.3 · p. 19

7 Conclusion

• Byteification produces fully open Bolmo models at the 1B and 7B scales that are on par with or surpass state-of-the-art subword models.The models come close to matching subword-level inference speed.Source: §7 · p. 19
• Byteification lets researchers adapt post-trained models in zero-shot once the corresponding base model has been byteified.It also permits decreasing token granularity for faster inference.Source: §7 · p. 19
• For the GPT3-Large matched Transformer baseline, counting input embeddings overestimates subword-model FLOPs by up to ∼25%.The passage attributes this to counting embedding computation that can instead be performed by a fast lookup.Source: §7 · p. 19

8 Future Directions

• Test whether byteification-oriented architectures, especially non-causal patch boundaries, also perform well when trained from scratch.The paper has restricted its evaluation to byteification and says the benefit of non-causal boundaries in scratch training is unknown.Source: §8 · p. 20
• Explore end-to-end learning of non-causal boundaries, including during post-training or by adapting methods that enable gradient flow through the predictor.The current boundary predictor was trained with direct external supervision.Source: §8 · p. 20
• Jointly increase local-model capacity and patch size, and investigate multi-byte prediction as potential routes to a better performance–efficiency tradeoff.Bolmo predicts only the next byte; even saving sequential local-model computations could yield speedups.Source: §8 · p. 20
• Investigate less destructive continued training, such as PEFT methods, to reduce the remaining performance gap with the source model.The authors attribute much of that gap to the continued-training setup generally hurting performance.Source: §8 · p. 20
• Study whether subword-level sampling methods transfer to LTLMs and develop byte-position-aware sampling strategies.The paper has not investigated how existing sampling methods transfer to LTLMs.Source: §8 · p. 20
• Assess alternative atomic input units because UTF-8 bytes are Latin-centric and it remains unclear how much latent tokenization can amortize that choice.The authors suggest investigating alternatives such as MYTE or SCRIPT.Source: §8 · p. 20
• Develop batched LTLM inference for wider applications; dynamic token and byte counts create additional optimization challenges.The reported competitive throughput is for batch size 1, which the authors say suffices for edge applications.Source: §8 · p. 20

A Additional Ablations

• Fused non-causal prediction of patch ends performs best among the tested boundary-predictor choices for byteification.Table 3 compares the alternatives.Source: A Additional Ablations · p. 29
• Using original subword boundaries as an oracle indicates that remaining boundary-predictor errors mostly explain the post-Stage 1 gap to the source model.The predictor still makes a small percentage of errors.Source: A Additional Ablations · p. 29
• Continued training without byteification generally degrades performance, although character understanding improves on targeted training data and remains below Bolmo.The passage suggests narrower data mix and suboptimal training as possible reasons for degradation.Source: A Additional Ablations · p. 29

• OLMES is used for all evaluations.Source: B Benchmark Details · p. 29
• Table 5 details the Bolmo 7B evaluation suite.Source: B Benchmark Details · p. 29
• Table 6 details the Bolmo 1B evaluation suite.Source: B Benchmark Details · p. 29

• About 75M tokens, roughly 0.04% of training data, cover spelling, reversal, swapping, deletion, and substitution tasks using 150,000 words with no overlap with CUTE test words.The character-task data is purely English.Source: C CUTE-Style Training Data · p. 29
• Large improvements on multilingual EXECUTE follow training with English-only character tasks, despite no multilingual character-understanding data.The authors report that the short training schedule otherwise does not impart this knowledge to byte-level models.Source: C CUTE-Style Training Data · pp. 29–30
• Table 4 attributes a substantial part of performance degradation to continued training of the source model, not specifically to byte conversion.Source: Table 4 · p. 30
• Longer training, more diverse data, or larger local models may be alternative routes to character-level knowledge.Source: C CUTE-Style Training Data · p. 30

• Embedding resettability is a prerequisite for transferring the global model into post-trained space via Task Arithmetic, because local models lack corresponding parameters in the post-trained checkpoint.Source: D Does Post-Training Byteified Models… · pp. 30–31
• Resetting embeddings preserves performance without substantial degradation for a substantial fraction of analyzed models, and almost perfectly preserves performance for Olmo 3 RL-Zero models.The authors observe a weak trend toward larger models being more amenable to resetting.Source: D Does Post-Training Byteified Models… · p. 31
• Future work could investigate when Task Arithmetic-based post-training is possible and whether byteifiability can be restored without additional training for models that currently lack it.Source: D Does Post-Training Byteified Models… · p. 31

E Embedding Rank Analysis

• Embedding matrices contain substantial high-rank structure, making them difficult to approximate with lower-dimensional local models.Linear upprojection or padding imposes a rank limit set by local-model dimensionality; concatenation avoids a hard limit but may still constrain expressivity.Source: E Embedding Rank Analysis · pp. 31–32

F Full Hyperparameters

• Full architecture and training hyperparameters are provided in Tables 7 and 8, respectively.Source: F Full Hyperparameters · p. 32, Table 7 · p. 34, Table 8 · p. 35
• Resetting embeddings is possible without substantial performance degradation for many analyzed models, with a weak trend toward larger models being more amenable.The figure also relates model parameter count to the loss ratio after resetting embeddings; remaining variance is presumed to reflect post-training choices.Source: Figure 8 · p. 33

E Embedding Rank Analysis

• Qwen3-4B-Base input embeddings are a notable exception to the broad high-rank pattern and may be more amenable to a lower-dimensional local encoder.Source: E Embedding Rank Analysis · p. 31, Figure 9 · p. 33