PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, Ngai Wong

arXiv:2407.13623v3cs.CLcs.AI

TL;DR

Scaling-law research has largely focused on model parameters and training data, leaving vocabulary size out of view despite its variation across models. This paper develops three ways to predict optimal vocabulary size and tests them across compute settings. Its predictions consistently outperform commonly used vocabulary configurations at matched FLOPs.

  • Problem

    Scaling laws have mainly modeled compute, parameters, and training data while usually disregarding vocabulary size, leaving the compute-optimal vocabulary size an open question.

    Source: §1 · p. 1

  • Method

    The paper combines IsoFLOPs power-law fitting, derivative-based estimation, and a vocabulary-aware parametric loss fit to predict optimal vocabulary size.

    Source: §1 · p. 2

  • Results

    Suggested vocabulary sizes consistently outperform commonly used configurations at the same FLOPs budget across undertraining, compute-optimal training, and overtraining settings.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Vocabulary size should be considered jointly with model parameters and training data when allocating resources for effective scaling.

    Source: §1 · p. 2, §7 · p. 10

    The estimates assume training data is optimally distributed; for models trained on substantially more data, optimal vocabulary sizes may be larger than estimated.

    Source: §1 · p. 3

Abstract

from arXiv · show

Research on scaling large language models (LLMs) has primarily focused on model parameters and training data size, overlooking the role of vocabulary size. We investigate how vocabulary size impacts LLM scaling laws by training models ranging from 33M to 3B parameters on up to 500B characters with various vocabulary configurations. We propose three complementary approaches for predicting the compute-optimal vocabulary size: IsoFLOPs analysis, derivative estimation, and parametric fit of the loss function. Our approaches converge on the conclusion that the optimal vocabulary size depends on the compute budget, with larger models requiring larger vocabularies. Most LLMs, however, use insufficient vocabulary sizes. For example, we predict that the optimal vocabulary size of Llama2-70B should have been at least 216K, 7 times larger than its vocabulary of 32K. We validate our predictions empirically by training models with 3B parameters across different FLOPs budgets. Adopting our predicted optimal vocabulary size consistently improves downstream performance over commonly used vocabulary sizes. By increasing the vocabulary size from the conventional 32K to 43K, we improve performance on ARC-Challenge from 29.1 to 32.0 with the same 2.3e21 FLOPs. Our work highlights the importance of jointly considering tokenization and model scaling for efficient pre-training. The code and demo are available at https://github.com/sail-sg/scaling-with-vocab and https://hf.co/spaces/sail/scaling-with-vocab-demo.

1 Introduction

The paper asks how vocabulary size should scale with compute, addressing scaling laws that mainly model parameters and training data while overlooking vocabulary. It argues for predicting vocabulary size jointly with these factors and tests those predictions empirically.

1 Introduction

  • Scaling laws have largely omitted vocabulary size, despite LLMs of similar parameter counts using markedly different vocabularies, such as Llama2-7B at 32K and Gemma-7B at 256K.This variation motivates the question of compute-optimal vocabulary size.Source: §1 · p. 1
  • Vocabulary size trades improved tokenization fertility and broader concept coverage against greater rare-token underfitting risk, especially when training data is constrained.The paper therefore says the optimum should account for both training data and non-vocabulary parameters.Source: §1 · p. 2
  • The paper proposes three ways to predict optimal vocabulary size: IsoFLOPs power-law fitting, a derivative-based estimate, and a parametric loss-function fit.The parametric fit also handles undertrained and overtrained allocations, unlike the first two approaches, which target compute-optimal settings.Source: §1 · p. 2
  • Across the proposed approaches, optimal vocabulary parameters follow a power law and scale more slowly than non-vocabulary parameters, with γ ≈ 0.83.Most existing LLMs allocate fewer vocabulary parameters than predicted; estimates assume compute-optimal training-data allocation.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • Suggested vocabulary sizes consistently outperform commonly used configurations across undertraining, compute-optimal training, and overtraining at the same FLOPs budget.The validation uses 3B-parameter models.Source: §1 · p. 3

2 Preliminary

This section sets out standard scaling-law allocation and loss concepts, then adapts them to compare models with different vocabulary sizes. The adaptation separates vocabulary parameters, measures data in characters, and normalizes loss against a unigram baseline.

2.1 Scaling law

  • Scaling laws allocate a FLOPs budget across model parameters and training tokens, using IsoFLOPs model groups to fit optimal allocations.IsoFLOPs groups hold compute budgets constant while varying parameter and data configurations.Source: §2.1 · pp. 3–4

2.2 Scaling law with vocabulary

  • The paper separates total parameters into non-vocabulary and vocabulary parameters, representing the latter as Vd, and measures training data in characters rather than tokens.Token counts depend on vocabulary size, whereas character counts support comparisons across vocabularies.Source: §2.2 · p. 4
  • A fitted function of vocabulary size estimates tokenizer compression ratio, mapping training characters to token counts for comparison with existing scaling-law studies.The fit uses a quadratic function of log vocabulary size and is evaluated on tokenizers with sizes from 1K to 1024K.Source: §2.2 · p. 4
  • The unigram-normalized loss measures a context-aware language model’s probability improvement over a unigram model and remains consistent across vocabularies for fixed non-vocabulary components.Unlike ordinary language-model loss, it supports fair comparisons among models with different vocabulary sizes.Source: §2.2 · p. 4
  • At each FLOPs budget, an optimal vocabulary size minimizes loss, and that optimal size increases as the budget grows.At fixed loss, the figure also depicts a vocabulary size that minimizes FLOPs.Source: Figure 3 · p. 5

3 Analysis: Why the optimal vocabulary size is bounded by compute

Vocabulary size creates a compute-bounded trade-off: increasing it first improves loss through better tokenization, but eventually the gain diminishes and loss rises. The vocabulary size at the loss minimum increases with the FLOPs budget.

3 Analysis: Why the optimal vocabulary size is bounded by compute

  • At small vocabulary sizes, improved tokenization fertility lets a model learn more characters from a fixed number of tokens, improving performance.Source: §3 · p. 5
  • At very large vocabulary sizes, fertility gains diminish while limited data may leave expanded vocabulary parameters inadequately trained, reducing performance.Source: §3 · p. 5
  • At a fixed FLOPs budget, loss first decreases and then rises as vocabulary size grows, with the loss-minimizing vocabulary increasing at higher budgets.The analysis estimates loss at each budget by interpolating between observed FLOPs and loss points.Source: §3 · p. 5

4 Estimating the optimal vocabulary size

The section presents three complementary approaches for estimating compute-optimal vocabulary size. Together, they fit vocabulary scaling, estimate it from derivatives, and predict it through a loss formula, including for settings where parameters and training data are not equally scaled.

4.1 Approach 1: Estimating power laws via IsoFLOPs

  • IsoFLOPs analysis varies vocabulary size from 4K to 96K across six non-vocabulary-parameter groups and selects the minimum normalized loss at each FLOPs budget.The selected points are fitted as power laws for non-vocabulary parameters, vocabulary parameters, and training characters.Source: §4.1 · p. 6, Figure 4 · p. 6, Figure 5 · p. 6
  • IsoFLOPs fitting gives Nnv = 0.08*C^0.50, Nv = 0.20*C^0.42, and H = 6.42*C^0.50, so vocabulary parameters scale more slowly than non-vocabulary parameters.The fitted vocabulary-to-non-vocabulary exponent ratio is γ = 0.42/0.50 = 0.84.Source: §4.1 · p. 6

4.2 Approach 2: Derivative-based fast estimation

  • Approach 2 reverses the usual fixed-compute optimization, seeking the minimum FLOPs needed to reach a specified normalized loss through vocabulary allocation.Source: §4.2 · p. 7
  • Derivative-based estimation combines γ from the derivative with an empirically optimal small-model vocabulary, yielding γ = 0.83 and slower scaling of vocabulary than non-vocabulary parameters.Near-optimal compute allocation lets FLOPs serve as a proxy for scaling optimal vocabulary parameters.Source: §4.2 · p. 7

4.3 Approach 3: Parametric fit of loss formula

  • Approach 3 fits a vocabulary-dependent loss formula and finds the optimal vocabulary by minimizing loss under a FLOPs constraint.Unlike approaches focused on compute-optimal allocations, it can predict locally optimal vocabulary size when non-vocabulary parameters and training characters are not equally scaled.Source: §4.3 · pp. 7–8
  • Vocabulary parameters are typically underestimated in practical models when parameter and training-data scales are not assumed equal.Figure 6 compares popular LLMs' vocabulary parameters with Approach 3 predictions at their reported training-token counts; Gemma2-9B is the stated exception.Source: §4.3 · p. 8, Figure 6 · p. 8

5 Discussion

The predicted vocabulary allocation improves downstream performance over the common 32K baseline in compute-optimal, scarce-data, and excessive-data settings. The discussion also relates optimal vocabulary size to training-data availability and flags its inference-compute cost.

5 Discussion

  • The three approaches give closely aligned optimal-vocabulary predictions, while mainstream LLMs typically allocate fewer vocabulary parameters than predicted.The predictions assume compute-optimal allocation, with training data scaled equally with non-vocabulary parameters.Source: §5 · pp. 8–9, Table 1 · p. 8
  • At Nnv = 2.87B, models using the predicted optimal vocabulary outperform the 32K baseline across multiple downstream tasks under compute-optimal training.The comparison uses equally scaled training data; Table 2 reports zero-shot accuracy.Source: §5 · p. 9, Table 2 · p. 9
  • With scarce or excessive training data, adjusting vocabulary size alone yields better models than the 32K baseline across different FLOPs budgets.Table 3 compares models with Nnv = 2.87B.Source: §5 · p. 9, Table 3 · p. 9
  • With Nnv fixed at 302M, the empirically optimal vocabulary shrinks from 16K to 10K when data is scarce and grows from 16K to 24K with excessive data.Figure 7 shows the best vocabulary size generally increasing as training data increases.Source: §5 · p. 9, Figure 7 · p. 10
  • Larger vocabularies increase inference computational demands, so the authors recommend choosing vocabulary size for a given model size under optimal training-data allocation, even if overtraining occurs.The discussion distinguishes this recommendation from optimizing vocabulary size for a particular non-optimal training-data amount.Source: §5 · p. 9

6 Related work

Related work studies language-model scaling laws and the allocation of compute across parameters and data. The authors identify vocabulary size as an effect those laws have generally ignored.

6 Related work

  • The paper studies models pretrained from scratch on English corpora and evaluates validation loss and downstream performance.The related-work overview describes Transformers' language-modeling role and examples of downstream capabilities.Source: §6 · p. 10
  • Language-model scaling laws model performance as a power law of compute allocated to parameters or training data, and later work considers downstream performance, inference time, and data constraints.Kaplan et al. [30] and Hoffmann et al. [26] are cited for parameter/data scaling and equal compute-optimal scaling, respectively.Source: §6 · p. 10
  • Prior language-model scaling-law work has generally ignored the effect of vocabulary size.The authors situate their work among scaling-law research spanning language models and other domains.Source: §6 · p. 10

7 Conclusion

The conclusion argues that vocabulary size is a consequential part of language-model scaling and can be allocated optimally for a given compute budget. Three prediction approaches yield aligned scaling patterns, and tests up to 3B parameters find better performance than conventional vocabulary sizes at equal FLOPs.

7 Conclusion

  • Across three approaches, vocabulary parameters should scale more slowly than non-vocabulary parameters, yet remain important for performance.The paper predicts optimal allocation and tests it on models up to 3B parameters and varying training-data amounts.Source: §7 · p. 10
  • Models trained with a predicted optimal vocabulary outperform models with a conventional vocabulary size under the same FLOPs budget.The conclusion reports empirical verification across varying amounts of training data.Source: §7 · p. 10

A.1 The derivation of FLOPs w.r.t the vocabulary size for the Approach 2

The appendix describes numerical derivative-based procedures for identifying vocabulary-size optima: one minimizes FLOPs, while the other minimizes loss at fixed compute.

A.1 The derivation of FLOPs w.r.t the vocabulary size for the Approach 2

  • Approach 2 sets the FLOPs derivative with respect to vocabulary size to zero and numerically solves for the vocabulary size that minimizes FLOPs.At that optimum, the model can train the maximum number of characters for a given FLOPs budget.Source: §A.1 · p. 18

A.2 The derivation of loss w.r.t the vocabulary size in Approach 3

  • Approach 3 substitutes the fixed FLOPs budget into the loss, differentiates loss with respect to vocabulary size, and numerically solves for the zero-derivative optimum.The loss is expressed as dependent on vocabulary parameters given the non-vocabulary parameter count.Source: §A.2 · pp. 18–19

A.3 More visualizations for the analyses: Why the optimal vocabulary size is bounded by the compute

Embedding clustering at larger vocabulary sizes suggests a compute-bounded trade-off between lexicon coverage and sufficient training. Extending the tested range shows performance degrades beyond the optimal vocabulary size.

A.3 More visualizations for the analyses: Why the optimal vocabulary size is bounded by the compute

  • D_avg falls from 1.067 at V=4K to 1.011 at V=16K and 0.952 at V=64K, indicating more embedding clustering at larger vocabularies.The visual compares learned embeddings at these vocabulary sizes for a model with Nnv = 85M; the authors associate clustering especially among infrequent words with insufficient training under constrained FLOPs.Source: Figure 9 · p. 19, §A.3 · p. 19
  • Larger vocabularies can improve lexicon coverage but hinder sufficient updating of word embeddings, especially for low-frequency words.The authors describe the optimum as balancing lexicon coverage against sufficient embedding updates.Source: §A.3 · p. 19

A.4 Exploration of Larger Range of Vocabulary Sizes

  • Across vocabulary sizes from 0.5K to 512K at fixed Nnv = 33M, loss consistently worsens once vocabulary size exceeds the optimal configuration.The exploration extends the vocabulary range beyond the 4K–96K range used to fit the scaling laws.Source: §A.4 · p. 19, Figure 10 · p. 20

A.5 The Vocabulary-insensitive Metric: Lu and BPC

The appendix reports a strong linear association between unigram-normalized loss and BPC, and details the modeling and experimental choices used in the scaling-law analysis. It also explains why vocabulary parameters are separated from non-vocabulary parameters.

A.5 The Vocabulary-insensitive Metric: Lu and BPC

  • Lu and BPC have a strong linear relationship, with ρ = 0.9888 and fit error e = 0.0683.BPC reflects external-text compression, while Lu measures token prediction loss normalized by token frequency.Source: §A.5 · p. 20, Figure 11 · p. 20

A.6 More Explanations about Why We Separate Vocabulary Parameters from the total Model Parameters

  • Vocabulary parameters in embeddings and language-model heads are generally confined to one layer, unlike non-vocabulary parameters that can grow through depth and width.The authors suggest vocabulary size may need to grow alongside depth to keep vocabulary parameters growing in balance with non-vocabulary parameters.Source: §A.6 · p. 21

A.7.1 Setting of model architecture, vocabulary size and training characters

  • Experiments fix non-vocabulary parameter counts by model family, vary vocabulary size, and use the Llama architecture except for vocabulary size.Vocabulary sizes are selected as multiples of 128; Table 4 lists model architectures and corresponding training-character counts.Source: §A.7.1 · p. 21, Table 4 · p. 21

A.7.2 The relationship between non-vocabulary parameters and embedding dimension

  • The analysis takes embedding dimension d as given based on prior work, using its relationship with Nnv specified in Table 5.This choice is made to ease modeling the scaling laws while accounting for vocabulary size.Source: §A.7.2 · p. 21, Table 5 · p. 22

A.7.3 Training details

  • Training uses AdamW, bfloat16 mixed precision, a global batch size of 512, and 40GB Nvidia-A100 GPUs.The largest reported Nnv = 2870M experiments take about 120 hours on 64 GPUs over 500B training characters.Source: §A.7.3 · pp. 21–22

A.7.4 Fitting techniques

  • The fitting procedures use log-form parameterization, Huber loss and LBFGS for Approach 1, with interpolation used to cheaply add data points.Approach 2 fits a scaling proportion, while Approach 3 fits the vocabulary-dependent loss formula with constraints on its scaling factors.Source: §A.7.4 · pp. 22–23

A.8 Details of fitting tokens-character relationship function f(V )

  • The authors train 25 tokenizers across vocabulary sizes from 1,024 to 1,024,000, then fit f(V) using validation-set token counts.They use scipy.optimize.curve_fit to estimate the function parameters.Source: §A.8 · p. 23

A.9 Robustness of the tokens-characters relationship function f(V )

The proposed tokens-character relationship function predicts tokenization fertility across tokenizer types. Its quadratic form is accurate over the tested range, while very large vocabularies require a correction to preserve decreasing behavior.

A.9 Robustness of the tokens-characters relationship function f(V )

  • The proposed f(V) predicts tokens-character ratios for BPE, unigram, and word-based tokenizers; unigram fertility is close to BPE, while word-based tokenization uses more tokens per character.The paper attributes the similar unigram and BPE fertility to both using subword tokenization.Source: §A.9 · p. 23, Figure 12 · p. 23
  • The quadratic log-vocabulary function fits the tokens-character ratio with RMSE 1.5e-6 and R2 0.99, but turns upward beyond approximately 218K.Because tokenization fertility improvements sharply decay at sufficiently large vocabulary sizes, the authors cap V at 200K when calculating f(V).Source: §A.9 · pp. 23–24
  • For a 300B-parameter model with sufficient training data, the predicted optimal vocabulary size is no more than 400K.For predictions at extremely large V, the authors say tokenizer training with larger vocabularies could improve precision.Source: §A.9 · p. 24

A.10 Experimental verification on the fairness of the unigram-normalized language modeling loss

The appendix evaluates unigram-normalized loss against downstream-task performance, extends vocabulary predictions to Llama3, and situates the study among related work. It also discusses vocabulary size as a possible constraint on scaling byte-level models.

A.10 Experimental verification on the fairness of the unigram-normalized language modeling loss

  • Across models with varying vocabulary sizes, raw language-model loss correlates positively with downstream performance, while Lu shows the expected negative correlation.The authors say larger vocabularies can raise raw language-model loss despite better downstream performance, and use this result to justify Lu for fair comparisons.Source: §A.10 · p. 24, Figure 13 · p. 24

A.11 Prediction for Llama3

  • The authors predict that Llama3's 128K vocabulary may remain insufficient for its 70B and 400B models.Figure 14 presents predictions for multiple Llama3 model sizes and gives an optimal vocabulary prediction of 487K for Llama3-400B.Source: §A.11 · p. 24, Figure 14 · p. 25

A.12 More Related Work

  • The related-work discussion frames optimal vocabulary allocation for downstream performance under fixed compute as an under-explored question and suggests 256-token byte-level vocabularies may constrain larger models.The authors present proportional vocabulary increases as a possible requirement for successful scaling of language models.Source: §A.12 · p. 25

B.1 Limitations of our proposed approaches

This section discusses tradeoffs and validation limits of the proposed approaches, notes further vocabulary-aware, multilingual, multimodal, and societal considerations, and reports that no crowdsourcing or human-subject research was conducted.

B.1 Limitations of our proposed approaches

  • Approaches 1 and 3 require broad experimental runs and are constrained by data-point coverage, while Approach 2 depends on accurate FLOPs and token-character models and cannot independently allocate non-vocabulary parameters or training data.Approach 1 also requires substantial compute for fitting; Approach 2 nevertheless offers rapid numerical estimates for preliminary configuration and closely matches experimental scaling-law predictions.Source: §B.1 · pp. 25–26

B.2 Larger models and different architectures

  • Predictions are validated only up to 3B parameters in dense transformers, leaving generalization to much larger models and non-transformer architectures untested.The paper names the 400-billion-parameter Llama-3 as an example of larger models and calls testing other scales and architectures future work.Source: §B.2 · p. 26

B.3 Parametric function for the loss when considering the vocabulary

  • Existing parametric loss functions model model size and training data but omit vocabulary, whose effects on model parameters, token counts, and tokenization quality complicate prediction.Future work could test nonlinear loss functions across compute allocations and datasets to model these interactions under varied data conditions.Source: §B.3 · p. 26

B.4 Extensions to multilingual and multimodal scenarios

  • Multilingual extensions must account for linguistic diversity and competition among languages when allocating vocabulary, while multimodal work must determine vocabulary and compute allocations across modalities.The paper identifies visual codebook size for image and video tasks as an open question.Source: §B.4 · pp. 26–27

C Potential social impact

  • The authors describe potential benefits in lower-cost, lower-energy models and broader access, alongside misuse risks such as deepfakes, phishing, spam, and harmful content.They call for trustworthy models, monitoring, and collaboration among researchers, policymakers, and users to mitigate risks.Source: C Potential social impact · p. 27

NeurIPS Paper Checklist

  • Pre-training experiments lack error bars because repeating runs across models, vocabularies, training characters, and experiments would be prohibitively expensive.The paper reports downstream-performance standard deviations and confidence intervals for experiments, but not pre-training error bars.Source: NeurIPS Paper Checklist · p. 30

B.1 Limitations of our proposed approaches

  • The paper marks crowdsourcing and human-subject research as not applicable, stating that no such experiments or participants are involved.Accordingly, the checklist items about participant instructions and compensation, risks, and IRB approval are marked NA.Source: NeurIPS Paper Checklist · p. 33

Related papers

Loading 2407.13623v3…