PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning is Forgetting: LLM Training As Lossy Compression

Henry C. Conklin, Tom Hosking, Tan Yi-Chern, Julian Gold, Jonathan D. Cohen, Thomas L. Griffiths, Max Bartolo, Seraphina Goldfarb-Tarrant

arXiv:2604.07569v1cs.LGcs.AIcs.CLcs.ITmath.IT

TL;DR

Limited understanding of LLM representational structure constrains accounts of how models learn and what information matters for performance. The paper uses an information-theoretic compression framework to analyze training, finding links between compression and downstream performance across models and benchmarks.

  • Problem

    How LLM representational spaces are structured and what information they retain for performance remain limited areas of understanding.

    Source: §1 · p. 1

  • Method

    The paper applies an information-theoretic lossy-compression framework to LLM representations, using soft quantization and context back-off to estimate information.

    Source: §1 · p. 1, §3.1 · p. 5

  • Results

    Pre-training models approach the optimal-compression bound, and compression optimality correlates significantly with performance across six benchmarks and six open-weight model families.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Compression optimality and retained information provide representation-level signals that predict downstream performance across model families.

    Source: §1 · p. 2

    Entropy estimates are sample-relative rather than estimates of true latent entropy, using 10,000 C4 or Tulu samples and a maximum context length of 512.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Despite the increasing prevalence of large language models (LLMs), we still have a limited understanding of how their representational spaces are structured. This limits our ability to interpret how and what they learn or relate them to learning in humans. We argue LLMs are best seen as an instance of lossy compression, where over training they learn by retaining only information in their training data relevant to their objective(s). We show pre-training results in models that are optimally compressed for next-sequence prediction, approaching the Information Bottleneck bound on compression. Across an array of open weights models, each compresses differently, likely due to differences in the data and training recipes used. However even across different families of LLMs the optimality of a model's compression, and the information present in it, can predict downstream performance on across a wide array of benchmarks, letting us directly link representational structure to actionable insights about model performance. In the general case the work presented here offers a unified Information-Theoretic framing for how these models learn that is deployable at scale.

1 INTRODUCTION

The paper frames LLM training as lossy compression: models retain information relevant to their objectives while discarding other information. It applies information theory to training dynamics and reports that compression structure relates to downstream performance.

1 INTRODUCTION

  • The paper addresses limited understanding of how LLM representational spaces are structured and what information matters for performance.It frames this as a challenge for interpreting how and what models learn.Source: §1 · p. 1
  • The paper treats LLMs as lossy compressors that retain training-data information relevant to their objectives, offering an information-theoretic account of learning.The authors connect this account to a unified theory of representation learning.Source: §1 · p. 1
  • OLMo2 7B representations approach the optimal-compression bound during pre-training as next-token-prediction loss saturates.Figure 1 plots complexity against expressivity and shows loss against proximity to the bound; its estimates use 10,000 C4 samples.Source: Figure 1 · p. 2, §1 · p. 2
  • The analysis aims to relate whole-model representational structure to learning and generalization, providing actionable insights at LLM scale.The introduction positions this as an alternative to understanding large models only through their individual parts.Source: §1 · p. 2
  • Across six benchmarks and six open-weight model families, compression optimality correlates significantly with performance; preference information predicts performance across 47 LLMs (r = 0.76, p < 0.001).The paper also reports that models across five open-weight families converge near optimal compression.Source: §1 · p. 2

2 BACKGROUND & RELATED WORK

The background situates lossy compression within theories of learning and inference, then introduces rate-distortion and Information Bottleneck frameworks. It contrasts this theoretical account with interpretability approaches that often focus on behavior, probes, or model components.

2.1 LEARNING, INFERENCE, AND COMPRESSION

  • Compression connects learning and inference across humans and models; this paper studies lossy compression during LLM training, distinct from in-context lossless compression.The authors follow the information-theoretic tradition while noting that compression has no universally agreed assessment.Source: §2.1 · p. 3

2.2 RATE DISTORTION THEORY

  • Rate-distortion theory trades input information preserved by a representation against prediction distortion; the Information Bottleneck uses I(X; Z) and I(Y; Z) to define complexity and expressivity.Optimal compression preserves only input information relevant to predicting the target; the paper uses the simpler I(X; Z) = I(Z; Y) bound, which is generally looser than the true frontier.Source: §2.2 · p. 3
  • Information Bottleneck theory predicts fitting followed by compression, but prior findings question whether compression is universal or necessary for generalization.Prior work had not tested these dynamics in Transformers trained on complex tasks.Source: §2.2 · p. 4

2.3 INTERPRETING NEURAL NETWORKS

  • Behavioral and probing methods characterize outputs or information recoverable from representations, while mechanistic interpretability studies circuits; the paper applies rate-distortion theory at LLM scale.The authors identify a disconnect between scalable interpretability methods and established learning theory, and present their approach as applicable to models at any scale.Source: §2.3 · p. 4

3 METHODS

The methods estimate representation entropy and mutual information at model scale using soft quantization and conditional context approximations. They use these quantities to assess compression and information associated with preferences.

3.1 ENTROPY ESTIMATION

  • The estimator provides an efficient alternative to binning-based quantization, whose memory and resource requirements make it difficult to apply at LLM scale.The authors apply a differentiable soft-entropy estimator that has been shown to converge to true distribution entropy.Source: §3.1 · p. 5
  • The method soft-quantizes normalized representations using cosine-based soft assignments to sampled sphere points, then aggregates assignments into layer distributions for Shannon entropy.The figure illustrates normalization, sampling, soft assignment, and aggregation.Source: §3.1 · p. 5, Figure 2 · p. 6

3.2 MUTUAL INFORMATIONS & BACK-OFF

  • To estimate conditional information tractably, the method approximates sequence distributions with n-gram back-off up to quadgrams and summarizes proximity to the Information Bottleneck bound as optimality.The figure illustrates that a bigram estimate conditions the current-token embedding on preceding context, rather than computing entropy over two token embeddings.Source: §3.2 · pp. 6–7, Figure 3 · p. 7
  • Preference information is estimated by conditioning representations on preferred versus rejected continuations; entropy estimates use fixed samples and a maximum context length of 512.Token and context estimates use 10,000 C4 samples, while preference estimates use 10,000 Tulu samples; these are sample-relative estimates, not claims about true latent entropy.Source: §3.2 · p. 7

4 EXPERIMENTS

The experiments track OLMo2 training and compare open-weight models at their final checkpoints to study compression dynamics and downstream performance. They find that compression structure and the information retained relate to performance across benchmarks.

4.1 PRE-TRAINING APPROACHES OPTIMAL COMPRESSION

  • OLMo2 7B follows the Information Bottleneck’s two-phase trajectory: output information first rises, then input information compresses as next-token loss begins to saturate.The model approaches the bound on optimal compression during this training trajectory.Source: §4.1 · p. 8
  • Across token, bigram, trigram, and quadgram back-off, training shows similar expansion-and-compression patterns, with larger contexts approaching the bound more closely.At higher contextualization, both complexity and expressivity decrease; the authors hypothesize this reflects similar source and target distributions in language modeling.Source: §4.1 · pp. 8–9
  • The 7B and 32B models approach optimal compression, whereas the 1B model oscillates away from the bound during the later training phase.The authors suggest a parameter threshold may be needed for optimal compression at a given data complexity.Source: §4.1 · p. 9, Figure 5 · p. 10

A Wide Array of Open-Weight Models Converge Along the Bound

  • Open-weight models across families, hyperparameters, and training methods cluster near the compression bound at the end of training, suggesting similar information structures.The experiments focus on OLMo2 intermediate checkpoints and compare other open-weight models at end of training; additional Smol LM2 and Pythia checkpoint analyses follow a similar pattern.Source: §4 · p. 7, A Wide Array of Open-Weight Models… · p. 9

4.2 RELATING REPRESENTATION STRUCTURE TO PERFORMANCE

  • At every back-off level, closer proximity to the Information Bottleneck bound is positively related to performance, using C4 rather than evaluation data.The authors report that this links simpler representations at a given expressivity to generalization across the benchmarks considered.Source: §4.2 · pp. 10–11
  • The expressivity-to-complexity ratio significantly predicts aggregate performance across six benchmarks (r = 0.52, p < 0.001).The ratio indicates how close a model is to optimal compression.Source: §4.2 · pp. 9–10, Figure 6 · p. 11
  • Preference information significantly predicts downstream performance (r = 0.76, p = 0.001), while C4 compression predicts math, reasoning, and factual knowledge but not instruction following.Instruction following is instead predicted by preference information.Source: §4.2 · pp. 11–12
  • Across 47 models, less token information and more bigram, trigram, and quadgram information relate to better downstream performance.The authors also connect this pattern to larger models allocating more representation space to contextual information.Source: §4.2 · p. 10, Figure 7 · p. 12

5 CONCLUSION

The paper frames LLM learning as lossy compression and reports that open-weight models converge along the Information Bottleneck bound. It connects compression and retained information to downstream performance and offers a whole-model approach to interpretability.

5 CONCLUSION

  • LLMs can be understood as lossy compression, with open-weight models converging along the Information Bottleneck bound.The paper links each model’s compression optimality to downstream benchmark performance.Source: §5 · p. 12
  • Different models retain different information, including information about local context and human preferences.The paper reports that compression optimality and preserved information predict downstream performance.Source: §5 · p. 12
  • The proposed interpretability approach studies the model as a whole rather than focusing on individual circuits, heads, or final-layer embeddings.It frames whole-model learning and generalisation in terms of established theories of learning and compression.Source: §5 · p. 12

7 ETHICS STATEMENT

The experiments use publicly available datasets and pretrained models under their original licenses, and report no known ethical concerns. The detailed visuals show open-weight models broadly approaching the Information Bottleneck bound, with trigram and quadgram estimates closer to the frontier but quadgram estimates noisy.

7 ETHICS STATEMENT

  • The authors report using publicly available datasets and pretrained models under their original licenses, with no additional data collection and no known ethical concerns.To their knowledge, the datasets contain no personally identifiable information, and the work complies with their terms of use and the ICLR Code of Ethics.Source: §7 · p. 13

A OPEN-WEIGHTS MODELS, DETAILED VISUALS

  • The 75 open-weight models span different levels of complexity and expressivity but broadly approach the Information Bottleneck bound.Hue indicates compression optimality, or proximity to the bound.Source: Figure 8 · p. 17
  • Trigram and quadgram models lie closer to the frontier than token models, but sparse samples make quadgram estimates noisy and sometimes apparently beyond the bound.The caption attributes apparent crossings to noise and the models’ proximity to the bound.Source: Figure 9 · p. 18

B POST-TRAINING AND PREFERENCE INFORMATION

The paper examines how post-training relates to preference information in Llama and Gemma models. Post-trained Llama models consistently have more preference information, while the pattern is consistent only for the latest Gemma 3 release.

B POST-TRAINING AND PREFERENCE INFORMATION

  • Figure 10 plots preference information against post-training status and, separately, against compression optimality.Significance values from a paired permutation test appear above the first comparison.Source: Figure 10 · p. 19
  • Post-trained Llama models consistently have higher preference information than their pre- and mid-trained counterparts.The paper interprets this as consistent with pre-training supplying core semantic information that post-training augments with task-specific and preference information.Source: B POST-TRAINING AND PREFERENCE… · p. 19
  • For Gemma, post-trained models have greater preference information in the latest Gemma 3 release, but earlier releases show no significant pattern.Source: B POST-TRAINING AND PREFERENCE… · p. 19

C PREDICTING PERFORMANCE ON INDIVIDUAL TASKS

Across individual tasks, optimal compression of C4 predicts performance on math, reasoning, and factuality benchmarks, while IFEval tracks preference information. Additional pre-training timecourses show that the 1.7B Smol model has expansion and compression phases but does not approach the bound.

C PREDICTING PERFORMANCE ON INDIVIDUAL TASKS

  • Optimal compression of C4 predicts math, reasoning, and factuality performance—but not IFEval (r = 0.07, p = 0.631); IFEval tracks preference information (r = 0.39, p = 0.004).Source: C PREDICTING PERFORMANCE ON INDIVIDUAL… · p. 20
  • Preference information predicts all benchmarks, though this may reflect preference-data composition; Tulu preference examples also favor relevant or correct answers.The paper says this preference-data characteristic proves important across all benchmarks.Source: C PREDICTING PERFORMANCE ON INDIVIDUAL… · p. 20

D ADDITIONAL MODEL TIMECOURSES

  • The 1.7B Smol model follows expansion and compression phases but does not approach the bound; smaller variants show little meaningful compression, possibly because checkpointing starts late.The 1.7B Smol model was trained on 11 trillion tokens and performs comparably to the 1B OLMo2 model trained on 4 trillion tokens.Source: D ADDITIONAL MODEL TIMECOURSES · p. 20

Individual Tasks

The section examines benchmark relationships, training timecourses, and methods for estimating information in LLM representations. It uses context-conditioned estimates and dimensionality-calibrated entropy estimation to compare models and training stages.

Individual Tasks

  • Optimal compression on C4 predicts performance on math, reasoning, logic, and factuality benchmarks, but not instruction following.Figure 11 plots individual task accuracy against C4 optimality and preference information across 47 open-weight models.Source: Figure 11 · p. 21, Individual Tasks · p. 21
  • Pythia 1.4B appears to compress better late than OLMo 1B, while Pythia 6.9B keeps expanding and appears undertrained.Pythia models see about 7.5% of the data used to train OLMo 1B, and use the same data amount, batch size, and steps across sizes.Source: Individual Tasks · p. 24

E.1 ESTIMATOR HYPERPARAMETERS

  • The estimator uses 100 bins; its temperature is calibrated separately for each model’s representation dimensionality.The calibration matches the estimator’s maximum value, log m, to an upper bound on the smoothed KL estimation target.Source: §E.1 · p. 24, §E.1.1 · p. 24

E.2 APPROXIMATING THE INPUT DISTRIBUTION

  • Fixed-width token, bigram, trigram, and quadgram contexts approximate the input distribution; unseen contexts are omitted.Quadgrams are sparse and less robust, so broad model comparisons use token and bigram estimates, while size analyses focus on trigrams.Source: §E.2 · p. 25

E.3 APPROXIMATING THE OUTPUT DISTRIBUTION

  • Teacher forcing gives each embedding gradient information from predictions of following tokens, so the output includes the entire trailing context.Input and output estimates are matched at the same context width.Source: §E.3 · p. 25

E.4 ESTIMATING MUTUAL INFORMATIONS

  • Mutual information is estimated from embedding entropy and conditional entropy, with context-conditioned distributions formed by subsetting embeddings.Token, bigram, and wider labels refer to the preceding context width, not to the number of embeddings whose entropy is computed.Source: §E.4 · p. 26

E.5 CONDITIONAL MUTUAL INFORMATIONS AND THE RESIDUAL

The section separates information associated with different context widths, relates those proportions to performance, and explains the entropy estimator and Information Bottleneck bound. It also describes computational constraints and the bound’s numerical approximation.

E.5 CONDITIONAL MUTUAL INFORMATIONS AND THE RESIDUAL

  • The chain rule separates current-token information from information in preceding context; each context-width proportion is normalized by model entropy.Residual information is what remains after subtracting mutual information for the most granular category, quadgrams.Source: §E.5 · pp. 26–27

E.5.1 CONDITIONAL MUTUAL INFORMATIONS AND PERFORMANCE

  • Less token information and a higher proportion of local contextual information relate significantly to downstream performance across 47 models.The reported main-paper correlation uses token-level back-off, where lower complexity relates to performance.Source: Figure 15 · p. 27, §E.5.1 · p. 27

E.6 ON THE USE OF SHANNON ENTROPY

  • The authors discretize representations to use Shannon entropy, since differential entropy is unbounded and changes under linear transformations.They also cite the cost of fitting an unknown embedding distribution as a reason to discretize.Source: §E.6 · p. 28

E.7 SCALABILITY OF PRIOR WORK

  • The soft-entropy estimator supports fast, memory-efficient online estimates in one forward pass, but cosine similarity uses angular information only.Unlike clustering-based quantization, its bins can be computed without fitting clusters to the data first.Source: §E.7 · p. 28

E.8 THE INFORMATION BOTTLENECK BOUND

  • The Information Bottleneck bound traces optimal compression as the trade-off parameter varies, but its numerical computation is intractable at natural-language scale.The authors approximate the bound as linear until saturation; tokenizer convergence begins to fail above 50,000 tokens.Source: §E.8 · p. 29

F RELATING COMPRESSION TO TRAINING LOSS

The section relates OLMo2 7B’s C4 loss to its distance from the Information Bottleneck bound and tests whether observed training patterns persist across data distributions. It reports a consistent two-phase pattern across C4, Tulu, and MMLU.

F RELATING COMPRESSION TO TRAINING LOSS

G ESTIMATOR ROBUSTNESS

  • The robustness experiments vary estimator hyperparameters and data distributions.The soft-entropy estimator is applied in this context, but was not introduced by this work.Source: G ESTIMATOR ROBUSTNESS · p. 30

G.1 ROBUSTNESS TO DATA DISTRIBUTION

  • OLMo2 7B shows expansion followed by compression toward the bound across C4, Tulu, and MMLU; Tulu and MMLU have higher mutual information than C4.The authors suggest the difference may reflect that Tulu and MMLU are more domain-specific than C4.Source: §G.1 · p. 30

G.2 ROBUSTNESS TO NUMBER OF REFERENCE POINTS

The section checks sensitivity to reference-point count and to averaging mutual information by mean rather than expectation. Both analyses retain the two-phase trajectory, with some estimator-dependent differences.

G.2 ROBUSTNESS TO NUMBER OF REFERENCE POINTS

  • Using 50 rather than 100 reference points preserves the expansion-and-compression pattern, with small changes to mutual-information values.The 100-point estimate has slightly lower mutual information than the 50-point estimate.Source: Figure 17 · p. 31, §G.2 · p. 31

G.3 LANGUAGE IS A LONG-TAILED DISTRIBUTION — COMPUTING ‘MUTUAL INFORMATION’ WITH MEANS

  • Mean- and expectation-based estimates both show expansion followed by compression, but the mean is not true mutual information and may not obey the Information Bottleneck bound.The mean reflects mean mutual information per label rather than weighting by the token representations’ distribution.Source: Figure 18 · p. 32, §G.3 · p. 32

H.1 LICENSES FOR MODELS AND DATASETS

The appendix records licenses for the estimation and evaluation datasets and model families, then reports the estimator's computational complexity and run costs.

H.1 LICENSES FOR MODELS AND DATASETS

  • Tulu and C4 are licensed under ODC-By v1.0, while MMLU Pro is licensed under Apache 2.0.Source: §H.1 · p. 32
  • OLMo, Qwen, and Pythia code and models use Apache 2.0; Gemma and Llama use their respective licenses, and Aya/Command uses CC BY-NC 4.0.Source: §H.1 · p. 33

H.2 COMPUTE RESOURCE AND COMPLEXITY

  • The paper estimates results using about 3,600 H100 GPU hours.Encoding 10,000 samples from C4 and Tulu takes approximately 10, 40, or 70 minutes on 2, 4, or 8 H100 GPUs, depending on model size.Source: §H.2 · p. 33
  • The estimation procedure requires only a dot product and softmax, while most compute expense comes from the model forward pass, which depends on model size.Source: §H.2 · p. 33

Related papers

Loading 2604.07569v1…