Language Modeling Is Compression

Introduction

• The source coding theorem equates optimal expected message length with negative log2-likelihood, making likelihood maximization equivalent to minimizing coding length.Arithmetic coding repeatedly refines probability-weighted intervals to encode symbols; its optimal coding length makes compression performance depend on the probabilistic model.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Offline compression uses fixed pretrained parameters in-context, making context length a key limit on how many bytes Transformers can compress at once.Transformers compress only a few kilobytes at a time while requiring substantial compute; the compression perspective provides insights into foundation-model failure modes.Source: §1 · pp. 1–2

Contributions

• Accounting for model parameters makes excessive scaling worsen compression, with dataset size imposing a hard limit on useful model size.Scaling laws hold for compression, but storing model parameters in the compressed output changes the trade-off.Source: §1 · pp. 2–3
• 43.4% on ImageNet patches and 16.4% on LibriSpeech samples: Chinchilla 70B beats PNG (58.5%) and FLAC (30.3%), respectively, excluding model parameter size.These cross-modal results come from a model trained primarily on text.Source: §1 · p. 2
• The compression-prediction equivalence also enables compressors to serve as generative models, visually illustrating the underlying compressor’s performance.Source: §1 · p. 3
• Tokenization acts as pre-compression but generally does not improve compression performance; instead, it increases information within the context to improve prediction.Source: §1 · p. 3

Coding Distributions

• A coding distribution assigns consistent probabilities across sequence lengths, defining next-symbol probabilities through the ratio of a sequence’s probability to its prefix’s probability.Consistency requires each prefix probability to equal the sum of its one-symbol extensions, with probability one assigned to the empty string.Source: §2 · p. 3

Lossless Compression

• Lossless compression minimizes expected code length while preserving recoverability, with Shannon entropy setting the lower bound L ≥ H(ρ).Frequent sequences receive shorter codes and rare sequences longer codes; H(ρ) = E[−log2 ρ(x)].Source: §2 · p. 3

Arithmetic Coding

• Arithmetic coding produces nearly optimal lossless codes, connecting compression quality directly to predictive log-loss.With infinite precision, code length is within one bit of −log2 ρ(x1:n). Finite B-bit precision adds O(n 2^-B) bits, negligible for 32- or 64-bit arithmetic.Source: §2 · p. 3

Arithmetic Encoder

• The arithmetic encoder repeatedly partitions an interval according to conditional symbol probabilities, retains the observed symbol’s interval, and selects its shortest binary-representable number.Encoding starts with [0, 1); each subinterval’s width equals the previous interval’s width multiplied by the corresponding symbol’s conditional probability.Source: §2 · p. 3

Arithmetic Decoder

• The arithmetic decoder reconstructs symbols using the encoded number and coding distribution, repeatedly identifying the containing subinterval and continuing within it.Decoding starts from [0, 1) and repeats the interval selection for each successive symbol.Source: §2 · p. 4

Likelihood Maximization

• Minimizing cross-entropy is equivalent to likelihood maximization and minimizes expected compressed length under an estimated source model.An estimated model replaces true conditional probabilities with model probabilities in the summed negative log-probabilities. Foundation-model log-loss training therefore uses a maximum-compression objective when paired with arithmetic coding.Source: §2 · pp. 3–4

Compression-Based Sequence Prediction

• Any compressor can support sequence prediction by deriving probabilities from its code lengths.Define ρ(x1:n) = 2^(-ℓ_c(x1:n)); conditional probabilities follow as ρ(x_i | x_<i) = 2^(ℓ_c(x_<i) − ℓ_c(x_<i x_i)).Source: §2 · p. 4

Universal Coding

• Universal coding theoretically recovers the Solomonoff predictor, whose log-loss is bounded by any computable predictor’s log-loss plus its program length (Solomonoff, 1964a;b; Rathmanner & Hutter, 2011).Choosing Kolmogorov complexity as code length gives theoretical universal optimality. The predictor mixes programmable predictors with weights 2^(-ℓ_c(q)); its additive program-length bound is independent of sequence length, connecting optimal compression and prediction.Source: §2 · p. 4

3 EXPERIMENTAL EVALUATION

• The evaluation compares arithmetic-coded neural language models against general-purpose and modality-specific lossless compressors.Baselines are gzip, LZMA2, PNG, and FLAC. Neural models include enwik8-trained Transformers, pretrained Llama 2 (Touvron et al., 2023b), and Chinchilla (Hoffmann et al., 2022).Source: §3 · p. 4

3.1 DATASETS

• 1GB datasets spanning text, images, and audio are evaluated in 2048-byte chunks, with additional unchunked classical-compressor results to expose chunking’s performance penalty.ImageNet uses flattened grayscale 32×64 patches, while LibriSpeech supplies audio chunks; enwik8-trained Transformers are evaluated on enwik8 and enwik9, whose first 10% is enwik8. Classical-compressor headers are counted only once across chunks.Source: §3.1 · pp. 4–5

3.2 COMPARING COMPRESSION RATES

• Text-pretrained foundation models outperform the tested classical compressors in raw compression across text, images, and audio, whereas enwik8-trained Transformers perform well mainly on Wikipedia.The authors attribute foundation models’ cross-modal performance to in-context learning (Genewein et al., 2023). Including model storage substantially worsens adjusted compression on the 1GB datasets.Source: §3.2 · pp. 5–6, Table 1 · p. 6

3.3 OPTIMAL MODEL-DATASET SIZE TRADEOFF

• Each dataset has an optimal Transformer size: larger models improve adjusted compression on larger datasets but worsen it once parameter costs dominate.Offline compression counts float16 parameters at 2 bytes each; foundation models require datasets on the order of TBs or more for non-trivial adjusted compression. Prequential coding can improve overparameterized-network compression (Blier & Ollivier, 2018), but requires costly online training during encoding and decoding.Source: §3.2 · p. 5, §3.3 · p. 6, Figure 2 · p. 7

3.4 COMPRESSORS AS GENERATIVE MODELS

• Compressors become autoregressive generators by deriving next-byte probabilities from compressed-length differences, but good compression does not guarantee good generated samples.For candidate byte b, the method uses 2^(ℓ_c(x_<i) − ℓ_c(x_<i b)) and repeatedly appends sampled outputs. Errors can accumulate, and one-byte lookahead misses favorable multi-byte dictionary extensions in gzip; next-step-trained neural models avoid this specific bias.Source: §3.4 · p. 7
• Chinchilla 70B produces qualitatively more appropriate, less noisy image continuations than gzip, although both generate incoherent samples and Chinchilla’s quality degrades with continuation length.For each ImageNet image row, models receive the first 250 pixels and generate the remaining 250, treating rows independently—an acknowledged oversimplification of image statistics.Source: §3.4 · p. 7, Figure 3 · p. 8

3.5 SEQUENTIAL EVOLUTION OF IN-CONTEXT COMPRESSION

• Compression rates generally improve rapidly with sequence length, with Chinchilla 1B outperforming gzip and an enwik8-trained Transformer across all three modalities and tested sequence lengths.The authors interpret this improvement as learning data statistics in-context without gradient-based training, allowing foundation models to adapt within relatively short context windows (Genewein et al., 2023).Source: §3.5 · p. 8, Figure 4 · p. 8

3.6 TOKENIZATION IS COMPRESSION

• Larger token vocabularies improve compression for small Transformers but harm larger models, trading shorter sequences and greater contextual information against harder next-token prediction.The experiments compare lossless ASCII tokenization against byte-pair encoding vocabularies of 1K, 2K, 5K, 10K, and 20K tokens. Shorter sequences also reduce Transformers’ quadratic context-length computation.Source: §3.6 · p. 8

Prediction vs. Compression

• CTW, PPM, and PAQ8 compress through model mixtures or context matching.CTW mixes Markov-model predictions, while PPM builds distributions from contiguous character contexts; both use arithmetic coding. PAQ8 combines many predictors and also includes noncontiguous context matches.Source: §4 · p. 9
• Compressors also support classification and policy-value estimation.Jiang et al. (2023) paired gzip with k-nearest neighbours for NLP, while Jiang et al. (2022) first trained compressors on unlabeled data. Image compression used arithmetic coding with Student-distribution mixtures and Gaussian-process predictors.Source: §4 · p. 9

Compression With Neural Networks

• Concurrent work (Valmeekam et al., 2023) investigated offline lossless compression using arithmetic coding with Llama 2 (7B) (Touvron et al., 2023b).This places foundation-model compression alongside established neural compression approaches.Source: §4 · p. 9
• Neural predictive distributions support lossless compression through arithmetic coding or asymmetric numeral systems, with both separately trained and online approaches.Examples include arithmetic coding (Schmidhuber & Heil, 1994; 1996) and asymmetric numeral systems (Hoogeboom et al., 2019). Online methods train only on the stream being compressed.Source: §4 · p. 9

Compression Biases: Tokenization, Model Size, etc.

• The paper interprets tokenization techniques as pre-compressors that can significantly affect final compression rates when combined with neural models.The discussion connects this interpretation to tokenization work including Kudo & Richardson (2018), byte-pair encoding, BPE dropout (Provilkov et al., 2020), and subword regularization (Kudo, 2018).Source: §4 · p. 9
• The section connects generalization to compression when parameter code length is included, and notes parameter compression as another route to shorter codes.It relates generalization studies (Neyshabur et al., 2017; Mirchandani et al., 2023; Ge et al., 2023) to compression, while Cheng et al. (2017) investigated compressing model parameters.Source: §4 · p. 9

• Large language models are competitive with standard compressors on text and on image and audio data they were never trained on.Source: §5 · p. 9
• Arithmetic coding converts sequence models into compressors, while coding lengths convert compressors into predictors through probability distributions following Shannon’s entropy principle.Source: §5 · p. 9
• The compression perspective accounts for model size, unlike standard log-loss, revealing that optimal model size depends on dataset size and cannot scale without limit.Source: §5 · p. 9

A ARITHMETIC CODING

• Arithmetic encoding repeatedly partitions the current interval according to predictive probabilities and selects the portion corresponding to the next input token.Starting from [0, 1), observing the first token A selects [0, 0.45); subsequent partitions use predictions conditioned on preceding tokens.Source: A ARITHMETIC CODING · p. 14
• After processing the string, arithmetic encoding repeatedly bisects [0, 1) until a binary subinterval is fully contained within the final coding interval.The sequence of binary choices supplies the compressed output.Source: A ARITHMETIC CODING · p. 14
• 7 bits encode AIXI as 0101010 in the worked example, compared with the four bytes used for the original string.The binary interval [0.328125, 0.3359375) lies fully inside the final coding interval.Source: A ARITHMETIC CODING · p. 14

B.1 DATA MANIPULATION

• Neural models accept all ASCII characters but not every higher-valued byte, so byte sequences are first mapped into [0, 127].This input constraint applies to both the Transformers trained on enwik and Chinchilla.Source: §B.1 · p. 14
• 1 bit is added to the final compressed sequence for each character requiring processing, and compression ratios account for this overhead.This prevents the ASCII conversion from artificially improving the reported compression ratio.Source: §B.1 · p. 14
• Text characters outside the ASCII range are mapped by clearing their most significant bit, which is appended to the compressed byte sequence.Source: §B.1 · p. 14
• 32-by-64 grayscale image patches become 2048-byte sequences when flattened, losing some two-dimensional pixel correlations.Each pixel occupies one byte; flattening places the last pixel of one row beside the first pixel of the next.Source: §B.1 · p. 14
• Image bytes are right-shifted into the ASCII range, with every byte transformed consistently and its removed least significant bit appended to the compressed output.The transformation applies even to bytes already below 128.Source: §B.1 · pp. 14–15
• 16 kHz LibriSpeech audio is reduced from two-byte int16 samples to one-byte samples, divided into 2048-byte chunks, and mapped to ASCII.As with images, every byte is right-shifted and its removed bit appended to the compressed output. Lowering the sampling rate did not significantly change relative model compression performance, so the original rate was retained.Source: §B.1 · p. 15

B.2 LARGE LANGUAGE MODELS TOKENIZATION

• 2048-byte inputs are compressed by arithmetic-coding token sequences using pretrained Transformer conditional probabilities, with compressed size compared against the original 2048 bytes.A tokenized sequence of length l yields l * T log-probabilities for an arithmetic encoder with vocabulary size T.Source: §B.2 · p. 15
• 32000-token vocabularies in Chinchilla and LLama2 transform 2048-character ASCII strings into variable-length sequences using SentencePiece (Kudo & Richardson, 2018), itself a form of lossless compression.The tokenized sequence length depends on the input.Source: §B.2 · p. 15
• 100 top next-token log-probabilities are renormalized at each step, almost fully recovering the conditional distribution while allowing arithmetic coding with a changing alphabet.Each step’s conditional probabilities must sum to 1. By contrast, the enwik-trained Transformers return full distributions and omit tokenization except in Section 3.6.Source: §B.2 · p. 15

Sampling limitations

• The simplest one-step autoregressive sampling setup produces biased gzip samples; looking multiple steps ahead would likely improve them.The authors retain this simple setup for all compressors rather than investigate improved sampling.Source: C ADDITIONAL RESULTS · p. 15

Text generation

• Chinchilla’s compression-based text continuations are significantly more coherent than gzip’s.The comparison uses 1948 bytes of enwik9 context and nominally samples 100 bytes; Chinchilla’s sampled tokens need not decode to exactly 100 bytes.Source: Figure C.1 · p. 16

Image generation

• Three differently sized Chinchilla models demonstrate compression-based image generation by autoregressively completing each image row.Each model receives the first 250 bytes of every row and samples the remaining 250 bytes.Source: Figure C.2 · p. 16

Audio generation

• Chinchilla’s audio predictions exhibit a typical autoregressive looping pattern.The audio experiment conditions gzip and Chinchilla on 1024 bytes from LibriSpeech and autoregressively generates the remaining 1024 bytes.Source: Figure C.3 · p. 17