PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

Zeyuan Allen-Zhu, Yuanzhi Li

arXiv:2404.05405v1cs.CLcs.AIcs.LG

TL;DR

The paper addresses the lack of a principled quantitative account of how much factual knowledge language models can store. It defines knowledge as tuples, measures storage through bit-complexity bounds in controlled datasets, and reports a universal 2bit/param capacity for sufficiently trained transformers, including int8-quantized models. The findings also characterize how training exposure, architecture, quantization, sparsity, and data quality alter this capacity.

  • Problem

    Existing scaling laws emphasize training requirements or benchmark performance, leaving the ultimate knowledge-storage capacity of sufficiently trained models quantitatively unresolved.

  • Method

    The paper generates synthetic knowledge tuples, trains language models of varying sizes, and compares trainable parameters with bit-complexity lower bounds for the stored knowledge.

  • Results

    A sufficiently trained transformer stores 2 bits of knowledge per parameter even when quantized to int8, while training duration, architecture, quantization, sparsity, and data quality affect capacity.

  • Takeaways & Limitations

    The framework provides a principled basis for comparing model architectures, training techniques, and data quality by knowledge-storage capacity rather than loss or benchmarks alone.

  • Takeaways & Limitations

    The study uses a restricted tuple-based knowledge domain and reports that further training may not improve capacity after 2bit/param, while other quantization methods are not covered.

Abstract

from arXiv · show

Scaling laws describe the relationship between the size of language models and their capabilities. Unlike prior studies that evaluate a model's capability via loss or benchmarks, we estimate the number of knowledge bits a model stores. We focus on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from a Wikipedia page. Through multiple controlled datasets, we establish that language models can and only can store 2 bits of knowledge per parameter, even when quantized to int8, and such knowledge can be flexibly extracted for downstream applications. Consequently, a 7B model can store 14B bits of knowledge, surpassing the English Wikipedia and textbooks combined based on our estimation. More broadly, we present 12 results on how (1) training duration, (2) model architecture, (3) quantization, (4) sparsity constraints such as MoE, and (5) data signal-to-noise ratio affect a model's knowledge storage capacity. Notable insights include: * The GPT-2 architecture, with rotary embedding, matches or even surpasses LLaMA/Mistral architectures in knowledge storage, particularly over shorter training durations. This arises because LLaMA/Mistral uses GatedMLP, which is less stable and harder to train. * Prepending training data with domain names (e.g., wikipedia.org) significantly increases a model's knowledge capacity. Language models can autonomously identify and prioritize domains rich in knowledge, optimizing their storage capacity.

1 Introduction

The paper introduces a controlled framework for measuring language-model knowledge storage in bits per parameter rather than through loss or benchmark performance. It finds a 2bit/param capacity under sufficient training, while training duration, architecture, quantization, sparsity, and data quality affect storage efficiency.

  • Framework: The framework defines knowledge as (name, attribute, value) tuples and estimates storage using bit-complexity lower bounds compared with trainable parameters.Synthetic knowledge-only datasets enable controlled variation of knowledge pieces and model sizes.
  • Core capacity: 2bit/param is achieved consistently by sufficiently trained GPT2 models across model sizes, architectures, data settings, and hyperparameters.The result holds across synthetic and semi-synthetic data and varied tuple characteristics.
  • Core capacity: A sufficiently trained 7B model is estimated to store 14B bits of knowledge, exceeding the combined knowledge of English Wikipedia and textbooks by the paper’s estimate.Stored knowledge is flexibly extractable through question answering and usable in downstream tasks after fine-tuning.
  • Training duration: Training exposure controls capacity: 1000 exposures are required for 2bit/param, whereas 100 exposures reduce undertrained GPT2 capacity to 1bit/param.The paper characterizes rarely encountered knowledge as being stored at the lower ratio.
  • Architecture: At 100 exposures, LLaMA/Mistral capacity is 1.3x lower than GPT2’s, and controlled experiments attribute the underperformance to gated MLP usage.With 1000 exposures, however, all tested models closely approach the 2bit/param rule.
  • Implications: The framework offers a controlled alternative to loss and benchmark comparisons, reducing confounds from architecture, data, and benchmark contamination.Its synthetic setting supports more precise comparisons of model capacity and training factors.
  • Quantization, sparsity, and data quality: Int8 quantization preserves capacity, whereas int4 reduces it to 0.7bit/param; MoE models lose only 1.3x capacity despite using 8.8% of total parameters during inference.The paper also reports that junk data can reduce useful-knowledge capacity by 20x, while prepending a domain token improves that loss factor to 2x.

2 Preliminaries

The paper represents knowledge as name–attribute–value tuples and constructs controlled synthetic and biography datasets to study how model size relates to knowledge capacity. It varies tuple complexity, dataset structure, and model/training settings to evaluate these scaling laws.

  • Knowledge representation: Tuple complexity depends on value length, vocabulary diversity, chunk count, and chunk diversity.For example, passport numbers encode more possible values than genders.
  • Dataset construction: The bioD dataset generates names, attributes, and value chunks randomly, with controllable numbers of names, attributes, tokens, chunks, lengths, and diversity.These parameters permit theoretical analysis of how knowledge complexity affects capacity.
  • Dataset construction: The study evaluates synthetic bioS, fixed-template bioSsimple, realistic bioR, and bioD datasets, alongside human biography data.bioS varies sentence-template selection and ordering online, whereas bioSsimple fixes them and bioR rewrites biographies with LLaMA2.
  • Training setup: The experiments vary dataset scale and exposure, including bioS datasets up to 20M individuals and bioR datasets up to 1M.Exposure counts knowledge-piece presentations rather than necessarily full passes through the data.
  • Training setup: Models use GPT2 with rotary positional embeddings, are trained from scratch with autoregressive loss, and typically use AdamW with fp16.The architecture sweep later includes LLaMA and Mistral.

3 Bit Complexity Lower Bound

The paper derives a bit-complexity lower bound for storing randomly generated tuple knowledge, accounting for name generation and value prediction losses. The bound is designed to match an information-theoretic upper bound in the ideal no-error case.

  • Evaluation target: Average token loss is insufficient for knowledge evaluation, so the analysis sums loss specifically over knowledge-bearing tokens.Filler phrases contribute to language modeling loss but not necessarily useful factual knowledge.
  • Evaluation target: The model is evaluated through name generation and value generation functions, with cross-entropy losses for names, values, and first value chunks.These losses are computed from autoregressive token probabilities.
  • Lower bound: Theorem 3.2 provides a bit-complexity lower bound under a random-dataset distribution, rather than for one fixed dataset.Randomness prevents a model from hard-coding a particular dataset into its architecture without trainable parameters.
  • Lower bound: The lower bound includes separate contributions from names, values conditioned on names and attributes, and value diversity.Dropping any component can produce a suboptimal bound.
  • Tightness: In the no-error case, uniform name prediction and perfect value prediction make the lower bound asymptotically match the upper bound.The name loss becomes log N while the value losses become zero.

4 Capacity Ratio

The capacity ratio measures learned knowledge relative to model parameters and its theoretical maximum. Experiments report ratios of at least 2 after 1000 exposures and at least 1 after 100 exposures, while the study does not scale beyond its tested synthetic setting.

  • Capacity definition: The empirical capacity ratio compares learned knowledge, derived from the lower bound, with the model’s parameter count.Separate definitions are given for bioD and the reduced bioS setting.
  • Main result: 2 bits per parameter is the reported peak capacity ratio after 1000 exposures, independent of model depth or size.The corresponding peak ratio is at least 1 after 100 exposures.
  • Scope boundary: The experiments remain within synthetic datasets because larger-model training was costly and the authors saw no need to scale further.GPT2-20-16 on bioS(10M) required 8.5 days with 64 A100s, whereas GPT2-12-32 on bioS(20M) for 100 exposures took 2.4 days.
  • Capacity definition: A model’s capacity ratio cannot exceed its maximum ratio, with equality achieved by a perfect model.For a fixed dataset, increasing model size eventually yields no additional knowledge and drives the maximum ratio toward zero.

5 Base Scaling Laws

The paper develops controlled scaling laws for knowledge storage, finding peak capacity ratios near 2 bits per parameter across datasets and broad data variations. The stored knowledge remains flexibly extractable for downstream tasks.

  • Capacity bounds: A model size P ≥ B/1.8 is sufficient for datasets containing B knowledge bits when the maximum capacity ratio is at most 1.8.The accompanying result also reports that no model exceeds capacity ratio 2.3.
  • Knowledge extraction: Knowledge learned at the reported capacity is flexibly extractable through fine-tuning and can support downstream comparisons and calculations.The paper distinguishes this from word-by-word memorization and verifies extractability in an appendix.
  • Data diversity: Rewriting data into diverse descriptions does not hurt capacity and may improve it, whereas low-diversity data wastes capacity memorizing sentence structures.GPT2 models trained on bioSsimple and bioR reach approximately 2 peak capacity ratios, though slightly lower, while the paper attributes the low-diversity loss to sentence-structure memorization.
  • Base scaling law: 2 bits/parameter is the recurring peak capacity ratio for GPT2 across broad bioD hyperparameter ranges.The result holds while varying K, C, D, L, and T across wide ranges.

6 Training Time vs Scaling Law

Training duration affects whether models reach peak storage capacity: 100 exposures still produce substantial capacity, but can lose up to a factor of 2 relative to 1000 exposures.

  • 100-exposure regime: R(F) ≥1 for GPT2 models trained for 100 exposures on bioS(N) across N from 10K to 10M and model sizes from 1M to 0.5B parameters.The result holds across a broad range of GPT2 model sizes and dataset sizes.
  • Training duration: 2x is the maximum reported capacity loss from using 100 rather than 1000 exposures.The paper states that 1000 exposures may be needed for maximum storage capacity, while 100 exposures remain within a factor of 2.

7 Model Architecture vs Scaling Law

Architecture has little effect after sufficient training, but differences emerge with only 100 exposures: gated MLPs reduce capacity, while moderately smaller MLPs do not.

  • Sufficient training: 1000 exposures make LLaMA, Mistral, and GPT2 architectures follow nearly the same scaling law.Removing the MLP entirely also leaves the capacity ratio broadly unchanged in this regime, aside from minor effects in tiny models.
  • Insufficient training: 1.3x worse capacity is reported for LLaMA than GPT2 in the 100-exposure setting, with similar results for Mistral.The comparison remains after optimally tuning learning rates and applies even to large models.
  • MLP design: Removing all MLP layers decreases capacity by more than 1.5x.This differs from reducing MLP size by one quarter, which has negligible impact.
  • Tokenizer effects: For tiny LLaMA models, matching GPT2 performance additionally requires the GPT2Tokenizer.The tokenizer effect is described as minor and specific to the biography data considered.
  • MLP design: Gated MLPs reduce capacity in insufficiently trained models, whereas reducing GPT2’s MLP size by 1/4 has negligible impact.The paper attributes the gated-MLP effect to lower capacity ratios and contrasts it with moderate MLP reduction.

8 Quantization vs Scaling Laws

Post-training quantization preserves most knowledge capacity at int8 but substantially reduces it at int4. Thus, int8 models can retain the reported 2-bit-per-parameter capacity, whereas int4 may require quantization-aware training.

  • Quantization effects: Int8 quantization has a negligible impact on the capacity of models trained with 16-bit floats.The reported experiment uses GPT2 models and post-training quantization.
  • Quantization effects: Int4 quantization reduces capacity by more than 2x.The paper reports this loss when reducing models from int8-level precision to int4 with the evaluated package.
  • Capacity limit: 2 bits/parameter remains attainable with int8 even at peak capacity.The authors state that extending training may not improve capacity further, while quantization can reduce it.
  • Theoretical bound: GPT2 can exceed one quarter of the absolute theoretical knowledge-storage limit for int8-based models.The paper derives this from the int8 upper bound R(F) ≤8.

9 Mixture of Experts vs Scaling Laws

The study finds that MoE models can use nearly all total parameters for knowledge storage despite sparse activation. Relative to GPT2, the capacity penalty is modest and persists even under the sparsest routing tested.

  • MoE is nearly fully efficient for knowledge storage, leveraging its total parameters despite sparsity constraints.The result holds even with 32 experts and is described as independent of the effective parameter count used per token.
  • 1.3x lower peak capacity ratio occurs for GPT2-MoE than GPT2 with 1000 exposures.
  • 1.5x lower peak capacity ratio occurs for GPT2-MoE than GPT2 with 100 exposures.
  • The result holds in the sparsest tested routing configuration, with topk = 1 and cap factor = 2.Similar results are reported for topk = 2 with cap factor 1 or 2.
  • The observed MoE degradation does not come from knowledge storage capability, despite typical underperformance versus dense models with equal total parameters.

10 Junk Data vs Scaling Laws

Low-quality random data substantially reduces useful-knowledge capacity, even with more training, whereas highly repetitive data does not. Adding a special token to useful data largely mitigates this degradation and can support domain-based prioritization.

  • 20x degradation in capacity ratio occurs when 7/8 of training tokens are junk and training remains at 100 exposures.The comparison is against training without junk at 100 exposures.
  • 3x, 1.5x, and 1.3x degradation remain at 300, 600, and 1000 exposures, respectively, with 7/8 junk data.These comparisons use the 100-exposure no-junk condition as the reference.
  • Highly repetitive junk data leaves the 100-exposure useful-data capacity ratio unchanged.This is the condition with 7/8 of tokens drawn from bioS(N′) for N′ = 1K.
  • Adding a special token to every useful example reduces the 100-exposure degradation to 2x and matches the 100-exposure no-junk scaling law at 300 exposures.The junk condition remains 7/8 of training tokens.
  • Domain names such as wikipedia.org can identify high-quality domains and help models prioritize their knowledge during training.The paper states that this identification requires no prior knowledge and is autonomous.

11 Conclusion

The paper develops controlled scaling laws for the number of knowledge bits stored by language models and finds a near-universal 2-bit-per-parameter capacity under sufficient training. Capacity depends strongly on training duration and data quality, while the knowledge remains flexibly extractable.

  • 2 bits of knowledge per parameter is the reported capacity of a sufficiently trained transformer, including after int8 quantization.The paper describes this as a universal scaling law and places it near the information-theoretical maximum.
  • Knowledge learned under the scaling law is flexibly extractable for question answering and downstream operations such as comparing retrieved facts.The paper evaluates extraction by fine-tuning with LoRA on question-answering texts and testing on held-out individuals.

A.3 Other Biography Datasets

Alternative biography datasets and architectures largely preserve the paper’s capacity scaling behavior, with modest reductions from rigid ordering or irrelevant textual detail. The 2bit/param pattern remains broadly universal across typical decoder-only architectures.

  • Other Biography Datasets: Fixed-order bioSsimple data slightly lowers capacity and produces nearly 0% extractability compared with diverse biography text.The paper interprets this as evidence that rewriting the same knowledge with greater data diversity helps extraction.
  • Other Biography Datasets: LLaMA2-generated bioR biographies slightly reduce capacity for larger models and more strongly for smaller models.The paper attributes this pattern to irrelevant details and the difficulty smaller models have with diverse English sentences.
  • Parameterized Scaling Laws: The bit-complexity lower bound is usually dominated by the value component, although name or diversity can become significant under some hyperparameter settings.
  • Parameterized Scaling Laws: Peak capacity ratios consistently exceed R(F) ≥2 when models below 50% accuracy are included.
  • Architectures: After 1000 exposures, LLaMA, Mistral, GPT20, and GPT21/4 closely follow GPT2’s scaling law, except for tiny models.Weight tying improves capacity in tiny models.

B.2 100-Exposure Setting

With 100 exposures, architecture affects how quickly models acquire knowledge rather than their eventual capacity: GatedMLP architectures train less stably, while standard MLPs can match GPT2’s scaling law.

  • Architecture comparison: 1.3x: LLaMA may lag behind GPT2’s scaling law even for larger models.
  • Architecture comparison: Replacing LLaMA’s GatedMLP with a standard MLP is necessary to match GPT2’s scaling law.
  • Architecture comparison: Mistral also underperforms GPT2, while reducing GPT2’s MLP size by a quarter has negligible impact and removing it significantly reduces capacity.
  • Training duration: The 100-exposure comparisons indicate differences in training ease, not that one architecture is strictly worse, because capacities become similar after 1000 exposures.
  • Training dynamics: GatedMLP slows learning and produces less stable training, whereas changing MLP size may have little effect.
  • Controlled changes: Changing LLaMA’s layernorm biases or activation from silu to gelu does not affect model capacity.

C More on Quantization

The section examines quantization and mixture-of-experts capacity under controlled training settings. Int8 preserves capacity, whereas int4 substantially reduces it, and MoE trades capacity for fewer inference parameters.

  • Quantization: Int8 quantization causes no change in capacity, while int4 quantization causes a capacity-ratio loss greater than 2x.
  • Quantization: For bioD data, 8-bit quantization has negligible impact, but 4-bit quantization reduces capacity by more than 2x, especially for large D and L.
  • Mixture of experts: MoE with 32 experts reduces capacity by 1.3x at 1000 exposures and 1.5x at 100 exposures, despite using 11.3x fewer inference parameters.
  • Mixture of experts: At 100 exposures, configurations (topk, cap factor) = (1, 2), (2, 1), and (2, 2) show minimal differences, with (2, 2) best for deep models such as GPT2-16-4.
  • Data composition: The experiments mix useful bioS(N) data with junk data, including random, highly repetitive, or specially prefixed alternatives.
  • Data composition: A special token at the front of useful data is analogous to adding a domain name such as wikipedia.org, which the model must autonomously recognize as useful.

F.2 Main Proof

The proof constructs random knowledge datasets and applies a bit-complexity lemma to relate model loss to the information required to encode names, attributes, diversity sets, and values.

  • Dataset construction: The proof applies Lemma F.1 by fixing ordered domains Q1 through QN and constructing the sampled dataset through sequential name and diversity assignments.
  • Dataset construction: The construction samples names, attribute-specific diversity strings, and tuple values uniformly without replacement or with uniform component sampling.
  • Prediction construction: For each name and attribute, the proof constructs predictions for name, diversity, and value components from the model outputs using fresh randomness.
  • Probability argument: A Chernoff bound shows that, when N ≥ Ω(D log N), each diversity value receives sufficient support with high probability.
  • Bound derivation: Summing the name, diversity, and value contributions and applying Lemma F.1 yields the theorem’s final loss and bit-complexity bound.
  • Knowledge comparison: The textbook estimate assumes no more than 100,000 books averaging 160,000 words, for approximately 16 billion words.
Loading 2404.05405v1…