PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

Zeyuan Allen-Zhu, Yuanzhi Li

arXiv:2404.05405v1cs.CLcs.AIcs.LG

TL;DR

How language-model size determines knowledge storage capacity remains insufficiently quantified. The paper represents factual knowledge as tuples and estimates stored bits using an accuracy-dependent lower bound. It reports a capacity of 2 bits per parameter across tested settings and architectures after sufficient training, even with int8 quantization.

  • Problem

    How knowledge storage capacity scales with model size lacks principled quantification, despite its importance for assessing transformer storage efficiency.

    Source: §1 · p. 2

  • Method

    The framework represents factual knowledge as (name, attribute, value) tuples and estimates stored bits using an accuracy-dependent bit-complexity lower bound.

    Source: §1 · pp. 2–3

  • Results

    2 bits of knowledge per parameter is the reported storage capacity across tested settings and architectures after sufficient training, even with int8 quantization.

    Source: §1 · pp. 3–4, §11 · p. 19

  • Takeaways & Limitations

    Knowledge-capacity comparisons can inform practitioners’ model selection and training-data preparation.

    Source: §11 · p. 20

    The study does not test further scaling to larger models in its synthetic setting, instead allocating GPU resources to other aspects of the study.

    Source: §4 · p. 9

Abstract

from arXiv · show

Scaling laws describe the relationship between the size of language models and their capabilities. Unlike prior studies that evaluate a model's capability via loss or benchmarks, we estimate the number of knowledge bits a model stores. We focus on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from a Wikipedia page. Through multiple controlled datasets, we establish that language models can and only can store 2 bits of knowledge per parameter, even when quantized to int8, and such knowledge can be flexibly extracted for downstream applications. Consequently, a 7B model can store 14B bits of knowledge, surpassing the English Wikipedia and textbooks combined based on our estimation. More broadly, we present 12 results on how (1) training duration, (2) model architecture, (3) quantization, (4) sparsity constraints such as MoE, and (5) data signal-to-noise ratio affect a model's knowledge storage capacity. Notable insights include: * The GPT-2 architecture, with rotary embedding, matches or even surpasses LLaMA/Mistral architectures in knowledge storage, particularly over shorter training durations. This arises because LLaMA/Mistral uses GatedMLP, which is less stable and harder to train. * Prepending training data with domain names (e.g., wikipedia.org) significantly increases a model's knowledge capacity. Language models can autonomously identify and prioritize domains rich in knowledge, optimizing their storage capacity.

1 Introduction

Controlled synthetic-knowledge experiments find that sufficiently trained language models store about 2 bits of factual knowledge per parameter. Training exposure, architecture, quantization, sparsity, and junk data affect capacity, while domain-like prefixes mitigate junk-data losses.

Knowledge Pieces

  • Knowledge is defined as a (name, attribute, value) tuple, enabling controlled experiments on how storage scales with model size.Randomly initialized models learn English descriptions of uniformly sampled tuples, primarily biographies produced through templates or LLaMA2-generated prose. Varying model and dataset sizes isolates knowledge-capacity scaling, while synthetic data remove irrelevant-data and benchmark-contamination concerns.Source: §1 · pp. 2, 3, 5

Bit Complexity and Capacity Ratio

  • Capacity ratio measures stored knowledge bits per trainable parameter using a bit-complexity lower bound that nearly matches the encoding upper bound.The lower bound measures the minimum bits required at the model’s achieved accuracy and matches the upper bound within a 1 - o(1) factor. This targets sufficiently trained capacity rather than training-time optimization or potentially confounded benchmark and perplexity comparisons.Source: §1 · pp. 2–4

Base scaling law for GPT2

  • 2 bits per parameter is consistently achieved by sufficiently trained GPT2 models across varied model sizes, architectures, datasets, and knowledge-generation settings.This predicts that a sufficiently trained 7B model stores 14B knowledge bits, exceeding English Wikipedia and textbooks combined according to the authors’ estimate. Stored knowledge supports flexible question answering [3] and downstream applications through fine-tuning [4], rather than merely verbatim memorization.Source: §1 · p. 3

How training time affects model capacity

  • 1000 exposures per knowledge piece enable 2 bits per parameter, while 100 exposures yield only 1 bit per parameter in undertrained GPT2.Exposure counts encounters with each fact, not dataset passes, because one pass can repeat a fact many times. The 100-exposure result also characterizes storage efficiency for relatively rare knowledge.Source: §1 · p. 3

How model architecture affects model capacity

  • 1.3x lower capacity than GPT2 is observed for LLaMA/Mistral at 100 exposures, even with best-tuned learning rates.At 1000 exposures, all tested architectures—including models without MLP layers—closely approach 2 bits per parameter. Controlled experiments implicate gated MLPs, which impair moderately rare knowledge but not frequently encountered knowledge.Source: §1 · p. 4

How quantization affects model capacity

  • 0.7 bits per parameter remains after int4 quantization, whereas int8 preserves capacity even at the 2-bit-per-parameter boundary.Quantization uses GPTQ [10], and int8 models can exceed one-quarter of their theoretical knowledge-storage limit. Further training may not improve saturated capacity; comparisons among quantization methods remain unexplored.Source: §1 · p. 4

How sparsity (MoE) affects model capacity

  • 1.3x lower capacity than the base scaling laws occurs with 32-expert MoE models, despite activating only 8.8% of total parameters during inference.The authors therefore suggest that MoE’s performance disadvantage against dense models with the same total parameter count likely does not arise from insufficient knowledge storage.Source: §1 · p. 4

How junk knowledge affects model capacity

  • 20x lower useful-knowledge capacity results from a 1:7 useful-to-junk token ratio at 100 exposures; domain-like prefix tokens reduce this loss to 2x.The model autonomously identifies high-quality data without prior knowledge of valuable domains. Without prefixes, increasing useful-knowledge exposures to 300, 600, and 1000 reduces loss factors to 3x, 1.5x, and 1.3x, respectively.Source: §1 · pp. 4–5

2 Preliminaries

The paper represents factual knowledge as (name, attribute, value) tuples and characterizes its complexity through dataset size, value structure, and vocabulary diversity. Its experiments use controlled biography datasets and language models trained from scratch.

2.1 Knowledge (Theoretical Setting)

  • Knowledge complexity depends on more than fact counts: the theoretical framework also parameterizes vocabulary size, value length, and diversity.N counts distinct names, K counts a fixed set of attributes, and T is the token vocabulary size.Source: §2.1 · p. 5
  • Each value contains C chunks of length L, with chunks drawn from an attribute-specific diversity set containing D possibilities.Equal chunk lengths and shared diversity sets simplify the notation; the theoretical results also accommodate attributes with different diversity sets or value lengths.Source: §2.1 · p. 5
  • The bioD generator samples distinct names and attribute-specific diversity sets without replacement, then samples each value’s chunks uniformly from the corresponding set.The paper provides a bit-complexity upper bound for describing the generated knowledge set when candidate names, attributes, and the token vocabulary are given.Source: §2.1 · p. 6

2.2 Knowledge (Empirical Setting)

  • 1000 exposures mean 1000 sightings per knowledge piece, corresponding to 1000 bioSsimple passes but only 25 bioR passes.For bioR, 100 exposures correspond to 2.5 passes. BioS rarely repeats identical biographies because sentence templates and their ordering vary.Source: §2.2 · pp. 6–7
  • Empirical bioD renders each person’s attributes in one paragraph using a consistent sentence template and randomized sentence order.Its candidate-name pool matches bioS, containing 400 × 400 × 1000 possible names.Source: §2.2 · p. 7
  • The empirical study primarily uses bioS, with freshly randomized biographies, alongside fixed-biography bioSsimple and LLaMA2 [33]-rewritten bioR.These adapt Allen-Zhu and Li [3]'s six-attribute biographies; bioR provides 40 rewrites per person. Experiments extend bioS to 20M individuals and bioR to 1M, yielding 22GB for bioR.Source: §2.2 · pp. 6–7

2.3 Models and Training

  • GPT2 is modified to use rotary positional embeddings [7], disable dropout, and retain 64 dimensions per attention head.GPT2-ℓ-h denotes ℓ layers, h heads, and 64h dimensions; the default GPT2Tokenizer produces variable-length token sequences for names and most attributes. LLaMA/Mistral architectures [19] [32] are also used for architectural comparisons.Source: §2.3 · p. 7
  • Models train from random initialization using autoregressive loss on randomly concatenated biography paragraphs, separated by EOS tokens and segmented into 512-token windows.Default training uses AdamW and mixed-precision fp16, with moderately tuned learning rates and weight decays.Source: §2.3 · p. 7

3 Bit Complexity Lower Bound

The section develops a bit-complexity lower bound and evaluates knowledge through summed losses on knowledge tokens. In the no-error case, the bound asymptotically matches Proposition 2.3’s upper bound.

3 Bit Complexity Lower Bound

  • Knowledge evaluation sums token losses for names, first value chunks, and complete values rather than averaging word-level loss.These quantities come from autoregressive cross-entropy losses, regardless of tokenizer or value length.Source: §3 · pp. 7–8
  • Under N ≥ Ω(D log N), the bit-complexity lower bound sums three components; omitting any can make the bound suboptimal.The setup permits training on bioD(N, K, C, D, L, T) using any optimizer. The paper studies how model parameter counts compare with this bound.Source: §3 · pp. 7–8
  • In the no-error case, the lower bound asymptotically matches Proposition 2.3’s upper bound.This requires generating each name with probability exactly 1/N and generating values with 100% accuracy. Then lossname(Z) = log N and lossvalue(Z) = lossvalue1(Z) = 0.Source: §3 · p. 8
  • The lower bound must consider a distribution over datasets because a fixed dataset could be hard-coded into an architecture without trainable parameters.Source: §3 · p. 8
  • The proof handles imperfect learning, unavoidable name-generation loss even for perfect models, and dependencies between values, names, and diversity-set choices.For fixed names and N facts each uniformly chosen from [T], perfect learning requires log2 |W| ≥ N log2 T.Source: §3 · p. 8

4 Capacity Ratio

The empirical capacity ratio measures learned knowledge per model parameter. Peak GPT2 ratios reach at least 2 bits per parameter after 1000 exposures and 1 after 100 exposures, regardless of model depth or size.

4 Capacity Ratio

  • Theorem 3.2 motivates an empirical capacity ratio defined using model losses and parameter count.For bioD datasets, the definition uses lossname, lossvalue, and lossvalue1, ignoring lower-order terms.Source: §4 · p. 8
  • ≥2 bits per parameter at 1000 exposures and ≥1 at 100 exposures are achieved by peak GPT2 capacity ratios across model depths and sizes.Figure 1 evaluates GPT2 pretrained on bioS(N) using fp16 mixed precision.Source: Figure 1 · p. 9, §4 · p. 9
  • Appendix experiments report similar scaling on bioSsimple and bioR, persistence under int8 quantization, and full extractability of learned knowledge.Learned knowledge is calculated using the bit-complexity lower bound from Theorem 3.2.Source: §4 · p. 9
  • 8.5 days on 64 A100s were needed to train GPT2-20-16 on bioS(10M) for 1000 exposures.GPT2-12-32 on bioS(20M) took 2.4 days for 100 exposures. The authors deemed further scaling unnecessary in this synthetic setting and allocated GPUs to other questions.Source: §4 · p. 9
  • 8 bits per parameter bounds R(F) for 8-bit models, ignoring lower-order terms; R(F) ≤ Rmax(F), with equality for perfect models.For a fixed dataset, increasing parameter count cannot add knowledge, so Rmax(F) approaches zero as model size increases.Source: §4 · p. 9
  • ≈47.6 bits per person, excluding names, is the bioS knowledge content; its reduced capacity ratio omits the diversity term.The bioS definition uses name and value losses together with parameter count.Source: §4 · p. 9

5 Base Scaling Laws

GPT2 models achieve peak knowledge capacities near 2 bits per parameter after 1000 knowledge exposures across biography formats and parameterized synthetic datasets. Rewriting training data preserves or can improve capacity, while sufficient text diversity enables flexible knowledge extraction rather than only verbatim memorization.

5 Base Scaling Laws

  • After 1000 exposures on bioS(N), with N ranging from 10K to 10M, GPT2 models with 1M–0.5B parameters achieve peak capacity ratios between 2 and 2.3, irrespective of depth or width.Training uses mixed-precision fp16. The 1000 exposures present each piece of knowledge 1000 times, rather than making 1000 passes over the data.Source: §5 · p. 10
  • P ≥ B/1.8 parameters suffice for a dataset containing B knowledge bits, with models attaining approximately their maximum capacity when Rmax(F) ≤1.8.This establishes sufficient model size beyond merely identifying a peak achieved by selected models.Source: §5 · p. 10
  • 2bit/param knowledge is flexibly extractable through question-answer fine-tuning [3] and usable for downstream manipulation [4], rather than merely memorized verbatim.The authors attribute extractability to sufficient textual diversity in bioS and verify it in Appendix A.2.Source: §5 · p. 10

5.1 Data Formats — Diversity and Rewriting

  • Approximately 2 is the peak capacity ratio after 1000 exposures on both bioSsimple and bioR, slightly below bioS; rewriting can preserve or improve capacity.bioSsimple contains one biography per person, whereas bioR contains LLaMA2-generated, near-human biographies. Without diversity, models waste capacity memorizing sentence structures, making rewriting preferable to repeatedly training on identical text.Source: §5.1 · pp. 10–11
  • 40 LLaMA2 rewrites trained for 25 passes each retain nearly the same capacity ratio as bioS, without requiring a 40x larger model.Rewrites can contain differing hallucinations; the slight capacity reduction is attributed to irrelevant data introduced by LLaMA2.Source: §5.1 · p. 11

5.2 Parameterized Scaling Laws

  • 2 or more is GPT2’s peak capacity ratio R(F) across broad variations in the bioD dataset’s five hyperparameters.The tested ranges are K and C from 1–50, D from 10–10,000, L from 1–50, and T from 20–40,000. The displayed analysis excludes models with accuracies ≤50%.Source: Figure 2 · p. 11, §5.1 · p. 11, §5.2 · p. 12

6 Training Time vs Scaling Law

Training with 100 exposures yields peak capacity ratios of at least 1 and loses no more than 2x capacity, although reaching maximum storage capacity may require 1000 exposures.

6 Training Time vs Scaling Law

  • 100 exposures yield peak capacity ratios R(F) ≥ 1 across GPT2 models with 1M to 0.5B parameters on bioS(N), with N ranging from 10K to 10M.This finding holds across the tested dataset and model sizes.Source: §6 · p. 12
  • 100 exposures incur no more than 2x capacity loss, although 1000 exposures may be necessary to reach maximum storage capacity.Source: §6 · p. 12
  • Section 10 will examine how mixing extremely low- or high-exposure knowledge with standard-exposure knowledge affects capacity.Example exposure counts are 1 and 1M for the extremes, compared with 100 for standard knowledge; these settings are addressed in Results 10 through 12.Source: §6 · p. 12

7 Model Architecture vs Scaling Law

With sufficient training, the tested architectures broadly share a 2-bit-per-parameter knowledge capacity, and GPT2 with rotary embeddings and no dropout performs no worse than LLaMA/Mistral. Under insufficient training, gated MLPs and removing all MLP layers reduce capacity.

7 Model Architecture vs Scaling Law

  • 2 bits per parameter is the broadly shared capacity ratio at 1000 exposures; GPT2 with rotary embeddings and no dropout performs no worse than LLaMA/Mistral.The tested architectures closely follow GPT2’s scaling law in this sufficient-training regime.Source: §7 · pp. 12–13, Figure 3 · p. 13

7.1 Insufficient Training Regime and a Closer Comparison

  • For tiny models, weight tying improves LLaMA/Mistral capacity with sufficient training, while GPT2Tokenizer additionally helps LLaMA match GPT2 under insufficient training.The weight-tying gap concerns models below 10M parameters at 1000 exposures. The tokenizer finding at 100 exposures is biography-data-specific, reflecting differences in how years are tokenized.Source: §7 · p. 13, Figure 3 · p. 13, §7.1 · p. 14
  • More than 1.5x capacity-ratio degradation occurs when all MLPs are removed at 100 exposures, whereas removing them at 1000 exposures leaves capacity unchanged.Reducing GPT2’s MLP size by 1/4 has negligible impact in both regimes. Preserved capacity without MLPs under sufficient training suggests attention layers can also store knowledge.Source: §7.1 · pp. 13–14, §7 · p. 13, Figure 3 · p. 13
  • 1.3x worse capacity ratios can occur for LLaMA versus GPT2 at 100 exposures, even for large models with optimally tuned learning rates; Mistral behaves similarly.Architectural differences therefore become apparent in the insufficient-training regime.Source: §7.1 · pp. 13–14, Figure 4 · p. 14
  • The authors propose controllable biography experiments as a testbed for evaluating future architectural designs.Source: §7.1 · p. 14
  • Replacing LLaMA’s gated MLP with a standard MLP improves larger models’ capacity ratios under insufficient training, even while retaining silu.Gated MLPs are less stable to train and require more time. Switching silu to gelu or adding trainable layer-normalization biases does not noticeably affect capacity ratios.Source: §7.1 · pp. 14–15, Figure 5 · p. 15

8 Quantization vs Scaling Laws

Post-training int8 quantization preserves knowledge capacity, whereas int4 quantization more than halves it with the tested package. Layer-removal observations suggest knowledge is distributed across layers rather than stored independently within them.

8 Quantization vs Scaling Laws

  • 2 bits/param of peak knowledge capacity is retained when models trained with 16-bit floats are quantized to int8 after training.The best capacity ratio remained 2 bits/param even after 1,000 training exposures on high-quality data. The authors infer that extending training may not improve capacity, whereas quantization can.Source: Figure 6 · p. 15, §8 · pp. 15–16
  • More than 1/4 of the absolute theoretical knowledge-storage limit is achievable by language models such as GPT2.For an int8-based model, the absolute upper bound on the capacity ratio is R(F) ≤ 8.Source: §8 · p. 16
  • More than a 2x capacity reduction occurs from int8 to int4, suggesting high-quality int4 models may require quantization during training.The post-training experiments used auto gptq, a package inspired by GPTQ [10], on models trained with mixed-precision 16-bit floats.Source: §8 · pp. 15–16

8.1 Where Is the Knowledge Stored?

  • Knowledge appears compactly distributed across Attention and MLP layers rather than isolated in individual transformer layers.Near the capacity boundary, removing the final layer of an L-layer model can eliminate much more than 1/L of its total knowledge, suggesting interdependent storage.Source: §8.1 · p. 16

9 Mixture of Experts vs Scaling Laws

MoE models scale similarly to dense models in knowledge storage, leveraging their total parameters despite sparse activation. The reported storage efficiency suggests that MoE’s typical underperformance relative to equally sized dense models does not stem from knowledge storage capability.

9 Mixture of Experts vs Scaling Laws

  • 11.3x fewer parameters are used during inference than exist in total in the illustrated 32-expert model, yet knowledge storage leverages all parameters.Each expert uses a d → d → d MLP; topk = 1 activates one expert per token, while attention parameters remain active.Source: §9 · pp. 16–17
  • 1.3x and 1.5x decreases in peak capacity ratio versus GPT2 occur for 32-expert GPT2-MoE under 1000-exposure and 100-exposure settings, respectively, counting total parameters.These modest decreases support nearly full knowledge-storage efficiency despite sparsity.Source: Figure 7 · p. 17, §9 · p. 17
  • Knowledge-storage efficiency persists with topk = 1 and cap factor = 2, with similar results for both tested topk = 2 configurations.The topk = 2 configurations use cap factor = 1 or cap factor = 2.Source: §9 · p. 17
  • MoE’s typical underperformance relative to dense models with the same total parameter count does not arise from deficient knowledge-storage capability.Source: §9 · p. 17

10 Junk Data vs Scaling Laws

Random-knowledge junk sharply reduces useful-knowledge capacity, whereas highly repetitive data does not. Prefixing useful data with a special token mitigates this degradation and, with additional exposures, recovers the no-junk capacity ratio.

Special-token mitigation

  • 300 exposures with useful-data prefix tokens match the capacity ratio of 100 exposures without junk, despite 7/8 junk tokens.At 100 exposures, prefixing useful data limits capacity-ratio degradation to 2x relative to training without junk.Source: Figure 8 · p. 18, §10 · p. 19

Random-knowledge junk

  • 20x capacity-ratio degradation can occur at 100 useful-data exposures when 7/8 of training tokens are random-knowledge junk, versus training without junk.At 300/600/1000 exposures, degradation remains 3x/1.5x/1.3x, respectively, compared with 100 exposures without junk. Random junk therefore harms useful-knowledge capacity even with substantially increased training.Source: Figure 8 · p. 18, §10 · p. 19

Repetitive data

  • 7/8 highly repetitive training tokens leave useful knowledge’s 100-exposure capacity ratio unchanged relative to training without junk.The repetitive-data condition uses bioS(N′) with N′ = 1K and does not impair useful-knowledge learning speed.Source: Figure 8 · p. 18, §10 · pp. 18–19

Experimental cost

  • 80x longer training than the no-junk baseline makes the 1000-exposure junk-mixture experiment expensive to extend to larger useful datasets.The mixture contains 1/8 useful tokens and 7/8 junk tokens from bioS(N′), with N′ = 100M. Even matching 100 useful-data exposures requires 8x longer training than without junk.Source: §10 · pp. 17–19

Practical implications

  • Prepending domain names to pretraining data could improve knowledge capacity by allowing models to autonomously identify knowledge-rich domains and prioritize learning from them.The proposed approach requires no prior knowledge about which domains contain high-quality knowledge.Source: §10 · p. 19

11 Conclusion … A.2 Knowledge Memorization vs. Extraction

Sufficiently trained transformers store 2 bits of knowledge per parameter, even when quantized to int8, and the scaling laws largely extend to extractable knowledge. The appendices detail the GPT2 experiments, training conditions, and extraction tests supporting these findings.

11 Conclusion

  • 2 bits of knowledge per parameter is the reported universal capacity of sufficiently trained transformers, including models quantized to int8.Sufficient training means that training loss has plateaued.Source: §11 · p. 19
  • The knowledge-capacity framework compares architectures, training techniques, and data quality while examining training duration, architecture, precision, MoE sparsity, and signal-to-noise ratio.The authors position this framework as a basis for model selection, data preparation, and further research into required model sizes.Source: §11 · pp. 19–20

A More on GPT2 Scaling Laws

  • GPT2 is modified with rotary embeddings [7] and dropout removal [33], then evaluated across wide-shallow and narrow-deep configurations.Experiments cover bioS(N) from 10K to 20M individuals; bioS(20M) contains approximately 1B knowledge bits. Models use 64 dimensions per head and a context length of 512.Source: A More on GPT2 Scaling Laws · pp. 20–21
  • 2 bits per parameter is validated at 1000 exposures using one near-capacity model per dataset for bioS(N) with N = 2M, 5M, 10M, and 20M.This restricted model selection conserves computation. Parameter counts exclude unused embedding tokens: bioS uses 3275 tokens, so GPT2small is counted as 88M rather than 124M parameters.Source: A More on GPT2 Scaling Laws · pp. 20–21

A.1 Base Scaling Laws

  • Mixed-precision fp16 and bf16 produce nearly identical GPT2 results.At 1000 exposures, final performance is relatively insensitive to learning-rate choices; at 100 exposures, careful learning-rate tuning is required.Source: §A.1 · pp. 21–22

A.2 Knowledge Memorization vs. Extraction

  • Extraction follows the framework of [3], using LoRA [17] question-answer fine-tuning on half the individuals and testing knowledge about the remaining half.Both accuracy metrics average five attributes per person, excluding company city; held-out extraction scores are summed and doubled to match the N-person memorization scale. LoRA ranks are varied across model sizes, and only the best results are reported to assess maximum extractability.Source: §A.2 · pp. 22–23
  • 1.2x lower total accuracy occurs only at the capacity boundary; GPT2’s bioS(N) scaling laws otherwise largely extend from memorized to extractable knowledge.The decrease concerns accuracy, not knowledge bits; extractable-bit measurements are omitted.Source: Figure 10 · p. 23, §A.2 · p. 23

A.3 Other Biography Datasets … B.1 1000-Exposure Setting

At 1000 exposures, typical decoder-only architectures closely follow GPT2’s 2bit/param capacity scaling, with weight tying helping tiny models. Biography diversity and irrelevant generated details affect capacity, while parameterized experiments show why knowledge accounting must include value, name, and diversity components.

A.3 Other Biography Datasets

  • Rewriting biographies with diverse templates increases knowledge capacity relative to fixed-order, fixed-sentence bioSsimple data.Allen-Zhu and Li [3] found that fixed-format knowledge was memorizable but nearly 0% extractable; diverse rewriting also improves extraction.Source: §A.3 · p. 24
  • LLaMA2-generated bioR biographies slightly reduce capacity ratios for larger models, with more pronounced reductions for smaller models.Generating each person’s biography 40 times produces 22GB of text. Irrelevant generated details consume capacity, while smaller models may also struggle to comprehend diverse English sentences.Source: §A.3 · p. 24

A.4 More on Parameterized Scaling Laws

  • Value usually dominates the knowledge-bit lower bound, but name and diversity contributions can be significant, motivating a bound that sums all three.The relative contributions depend on the experimental hyperparameter settings.Source: §A.4 · p. 25
  • R(F) ≥2 is consistently exceeded by peak capacity ratios in the full parameterized scaling results.These GPT2 experiments use bioD data with 1000 exposures, weight decay 0.01, learning rate 0.0005, and batch size 192.Source: §A.4 · pp. 24–25, Figure 12 · p. 25

B More on Model Architectures

  • LLaMA [32] [33] and Mistral [19] differ from rotary, dropout-free GPT2 through gated MLPs, untied embedding/output weights, tokenizers, and normalization.GPT2 uses layer normalization with trainable bias, unlike LLaMA/Mistral. Mistral uses group-query attention, while LLaMA favors multi-query attention only for very large models.Source: B More on Model Architectures · p. 27
  • Reduced-MLP and MLP-free GPT2 variants test whether MLP layers are primarily responsible for knowledge storage.The reduced variant changes the MLP from d → 4d → d to d → d → d, retaining one-quarter of its original size.Source: B More on Model Architectures · p. 27

B.1 1000-Exposure Setting

  • At 1000 exposures, except for tiny models, LLaMA, Mistral, GPT20, and GPT21/4 closely follow GPT2’s scaling law, supporting a relatively universal 2bit/param capacity ratio among most typical decoder-only language model architectures.Weight tying increases capacity for tiny models with ≤10M parameters. GatedMLP makes LLaMA and Mistral training less stable, necessitating mixed-precision bf16 training when required.Source: §B.1 · p. 28

B.2 100-Exposure Setting … F.1 Warmup Examples

This section examines learning speed, training stability, and knowledge capacity through comparisons of architectures, int8 quantization, and sparse MoE at 100 exposures. It also applies Lemma F.1 to derive model bit-complexity lower bounds for value-only and name-only memorization.

B.2 100-Exposure Setting

  • 1.3x is LLaMA’s possible shortfall versus GPT2’s scaling law at 100 exposures; replacing GatedMLP with standard MLP is necessary to match GPT2.GatedMLP slows and destabilizes learning despite selecting its best of three learning rates, versus one for standard MLP; smaller models additionally require weight tying and GPT2’s tokenizer. These differences concern insufficient training, since architectures achieve similar capacity ratios at 1000 exposures.Source: §B.2 · p. 28
  • Removing MLP layers substantially reduces capacity and slows learning, whereas resizing them has little effect; Mistral also underperforms GPT2 because of GatedMLP.Enabling trainable biases in LLaMA’s layernorms or switching silu to gelu does not affect capacity.Source: §B.2 · pp. 28–29

C More on Quantization

D More on Mixture of Experts

  • 11.3x fewer inference parameters accompany 32-expert MoE capacity reductions of 1.3x at 1000 exposures and 1.5x at 100 exposures versus dense GPT2.The strongest reported configuration uses topk = 1 and cap factor = 2 during training, increasing cap factor to 8 during testing.Source: Figure 16 · p. 32, C More on Quantization · p. 32, D More on Mixture of Experts · p. 32
  • MoE routing configurations differ minimally at 100 exposures, although topk = 2 and cap factor = 2 perform best for deep models.The comparison tests (topk, cap factor) = (1, 2), (2, 1), and (2, 2); MoE sparsity also motivates higher learning rates than dense models.Source: D More on Mixture of Experts · p. 32, Figure 17 · p. 33

E More on Junk Data vs. Scaling Laws

  • 1/8 of training tokens contain useful data and 7/8 contain junk; random-junk experiments extend useful-data exposure counts from 100 through 1000.Random junk uses bioS(N′) with N′ = 100M. Computational limits restrict 1000-exposure experiments to N ≤200K and 300- or 600-exposure experiments to N ≤500K.Source: E More on Junk Data vs. Scaling Laws · p. 33
  • The authors characterize repetitive-junk experiments as positive and prefixing useful data amid random junk as moderately positive.Repetitive junk uses bioS(N′) with N′ = 1K; the prefix is a special token before useful data. Negative-result experiments receive broader learning-rate searches and smaller batches, while positive-result experiments receive less optimization.Source: E More on Junk Data vs. Scaling Laws · pp. 33–34

F Proof of Theorem 3.2

  • Lemma F.1 relates the encoding bit complexity of independently sampled training variables to probabilities of correctly predicting their values.Predictors may use earlier data and fresh randomness. The proof combines counting with merging variables, while Jensen’s inequality supplies the second inequality.Source: F Proof of Theorem 3.2 · p. 35

F.1 Warmup Examples

  • For value-only prediction, the bit-complexity lower bound matches the upper bound as cross-entropy loss approaches zero.The setup predicts N independent, uniformly sampled values from [T], with cross-entropy averaged over possible training datasets.Source: §F.1 · pp. 35–36
  • The name-only proof represents names by ranks among remaining candidates, exploiting Lemma F.1’s allowance for predictions to depend on previously revealed names.The construction uses shrinking choice sets of sizes N0, N0−1, through N0−N+1.Source: §F.1 · p. 36
  • log N is the best possible loss when the name-only model perfectly memorizes its training set, yielding a bit-complexity lower bound.The training set contains N distinct names sampled uniformly without replacement from [N0].Source: §F.1 · p. 36

F.2 Main Proof · G Missing Remark

The proof of Theorem 3.2 constructs the dataset from independent random indices and combines name, diversity, and value bounds using Lemma F.1. A separate remark estimates English-language textbook content at 16 billion words under explicit book-count and length assumptions.

F.2 Main Proof

  • N distinct names are selected without replacement using independent, uniformly random indices in shrinking domains.The index domains have sizes N0, N0 − 1, through N0 − N + 1.Source: §F.2 · p. 36
  • Each attribute receives D distinct vocabulary elements, and each name–attribute value comprises C indexed elements from that attribute’s vocabulary.This alternative construction preserves the dataset distribution in Definition 2.2.Source: §F.2 · pp. 36–37
  • The name decoder uses fresh randomness and indexes generated names among remaining candidates, returning a failure symbol for previously selected names.Its analysis follows the name-only warmup example.Source: §F.2 · p. 37
  • N ≥ Ω(D log N) is the stated condition for the diversity argument’s Chernoff bound, with model outputs evaluated at uniformly sampled names.The diversity decoder indexes outputs among vocabulary elements not previously selected, returning a failure symbol when no match exists.Source: §F.2 · p. 37
  • The value decoder maps each predicted component to its index in the attribute vocabulary, returning a failure symbol if any component lacks a match.Successful decoding produces a sequence of C indices.Source: §F.2 · p. 38
  • The proof of Theorem 3.2 concludes by summing bounds (F.2), (F.3), and (F.4) and applying Lemma F.1.These bounds address the name, diversity, and value components.Source: §F.2 · pp. 37–38

G Missing Remark

  • 16 billion words is the estimated English-language textbook total, assuming content condenses into at most 100,000 books averaging 160,000 words.The remark acknowledges that substantial overlap, especially among PreK–12 textbooks, complicates this estimate.Source: G Missing Remark · pp. 38–39

Related papers

Loading 2404.05405v1…