PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws

Xu Ouyang, Deyi Liu, Yuhang Cai, Jing Liu, Yuan Yang, Chen Zheng, Thomas Hartvigsen, Yiyuan Ma

arXiv:2605.23901v1cs.LGcs.AIcs.IT

TL;DR

Existing monotonic scaling laws fail to explain U-shaped degradation associated with catastrophic overtraining and quantization. The paper proposes a Shannon Scaling Law that models LLMs as noisy channels by mapping model size to bandwidth, tokens to signal, and training effects to noise. The law captures these behaviors and extrapolates to an unseen 12B model and longer token range with pooled R2=0.847.

  • Problem

    Traditional power-law scaling laws fail to model U-shaped loss curves arising from catastrophic overtraining and quantization-induced degradation.

  • Method

    The Shannon Scaling Law models LLM training as information transmission over a noisy channel, mapping model size to bandwidth and training tokens to signal while modeling noise.

  • Results

    The law maintains pooled R2=0.847 when trained on ≤6.9B Pythia models with ≤180B tokens and tested on the unseen 12B model up to 307B tokens.

  • Takeaways & Limitations

    The results support treating LLM scaling through capacity and SNR rather than assuming performance improves monotonically with model size or data.

  • Takeaways & Limitations

    The loss-capacity formulation assumes loss approaches 0 as capacity approaches infinity and uses a reciprocal relationship between test loss and capacity.

Abstract

from arXiv · show

Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite increased compute. We propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem. By mapping model parameters to channel bandwidth and training tokens to signal power, our formulation explicitly captures the interaction between learning signal and intrinsic noise. This perspective reveals a fundamental Shannon capacity for LLMs: scaling model size or data without preserving a sufficient signal-to-noise ratio (SNR) inevitably amplifies noise, inducing a transition from monotonic improvement to U-shaped performance degradation. We validate our theory through experiments on Pythia and OLMo2 under perturbations, including Gaussian noise, quantization and supervised fine-tuning on math, QA and code tasks. The Shannon Scaling Law consistently outperforms classical scaling laws and recent perturbation-aware laws, achieving strong $R^2$ scores and accurately capturing loss basins missed by prior approaches. It also extrapolates: fitted on $\leq$6.9B Pythia models with $\leq$180B tokens, it predicts the unseen 12B model up to 307B tokens at pooled $R^2{=}0.847$, while monotonic baselines collapse.

1 Introduction

The paper argues that monotonic scaling laws miss degradation from overtraining and quantization, and introduces a Shannon-based noisy-channel framework to unify monotonic and U-shaped behavior. Its law models capacity through model size, tokens, and noise, while extrapolating beyond the fitted Pythia range.

  • Motivation: Scaling traditionally improves performance monotonically with compute, parameters, and dataset size, but recent findings challenge this assumption.The paper cites trillion-parameter models and massive corpora as products of the prevailing scaling hypothesis.
  • Motivation: Catastrophic overtraining and quantization-induced degradation produce U-shaped loss curves that traditional power laws fail to model.Performance initially improves but eventually deteriorates as scaling continues.
  • Shannon perspective: The proposed framework treats an LLM as a noisy channel, with pretraining modulating information into weights and inference transmitting information from context X to output Y.The analogy motivates applying the Shannon-Hartley theorem to bounded capacity under data and architectural noise.
  • Proposed framework: The Shannon Scaling Law maps model size to bandwidth, tokens to signal, and data, model, and interference effects to noise.Quantization perturbations enter the noise term, reducing effective capacity and producing U-shaped scaling behavior.
  • Results: Pooled R2=0.847 is achieved when fitting ≤6.9B Pythia models with ≤180B tokens and predicting the unseen 12B model up to 307B tokens.The framework outperforms existing baselines across quantization, SFT, and Gaussian-noise perturbation scenarios.

2 Preliminary and Related Works

The paper reviews monotonic power-law scaling and perturbation-aware extensions before motivating a noisy-channel interpretation grounded in Shannon capacity. This interpretation reframes LLM capacity using signal, noise, bandwidth, and SNR.

  • Scaling laws: Scaling laws quantify relationships between loss or perplexity and model parameters N and training tokens D.Traditional formulations generally assume performance improves monotonically as resources increase.
  • Scaling laws: OpenAI and Chinchilla laws use monotonic power-law formulations, with Chinchilla incorporating compute through an additive form fitted from optimal losses.Their parameters include fitted coefficients, exponents, and an irreducible-loss term.
  • Perturbation-aware laws: Perturbation-aware laws add degradation terms to base scaling laws to model U-shaped losses from quantization or overtraining.The reviewed formulations represent penalties associated with low-bit precision and excessively large models or data sizes.
  • Noisy-channel foundations: The Shannon-Weaver model represents communication as source, transmitter, noisy channel, receiver, and destination, paralleling the paper’s LLM analogy.Prior deep-learning work also characterizes networks through mutual information I(X; Y).
  • Noisy-channel foundations: The Shannon-Hartley theorem defines an upper bound on error-free information rate using bandwidth B, signal power S, noise power N, and SNR S/N.The paper reinterprets these physical quantities to model LLM representational capacity.

3 The Shannon Scaling Law

The Shannon Scaling Law defines LLM capability as a noisy-channel capacity, mapping model size to bandwidth and tokens to signal while explicitly modeling training noise. It links capacity reciprocally to loss and explains degradation through insufficient signal-to-noise conditions.

  • Capacity formulation: The paper defines CLLM as an upper bound on the rate at which knowledge can be learned and represented under a specific compute and data budget.This capability is introduced as an analogue of Shannon channel capacity.
  • Capacity formulation: The law extends channel capacity by mapping physical components B, S, and N to model size N and training tokens D.Its fitted positive constants parameterize the resulting formulation.
  • Component analysis: Model size acts as bandwidth because larger models provide more space to capture features and patterns.The paper models this relationship as a power law.
  • Component analysis: Training-corpus knowledge is treated as signal, with average signal power proportional to the number of training tokens D.The signal relationship follows established power-law scaling conventions.
  • Component analysis: The noise formulation includes data-induced noise, model-interaction noise, and irreducible noise from system entropy or architectural limitations.Data-induced noise accumulates with D=bs × t, while model-interaction noise changes over the training trajectory.
  • Capacity and loss: The proposed reciprocal capacity-loss relationship makes loss decrease toward 0 as capacity approaches infinity and increase sharply as capacity approaches zero.It also encodes nonlinear diminishing returns: later loss reductions require larger capacity increases.

4 Experiments

Experiments evaluate the Shannon Scaling Law across model families and perturbations, including Gaussian noise, supervised fine-tuning, and quantization. Across these settings, it models non-monotonic loss landscapes and generally outperforms competing scaling laws.

  • Experimental Setup: Experiments use Pythia and OLMo2 checkpoints, with wikitext2 test loss as the fitting target and Gaussian noise, SFT, and quantization as perturbations.SFT covers GSM8K, SiQA, and StarCoder-Python; quantization spans 16-bit to 2-, 3-, and 4-bit checkpoints.
  • Gaussian Noise: Under Gaussian noise, increasing noise breaks monotonic scaling and produces U-shaped loss curves along both model-size and token axes.At 10 dB, the low-loss region shrinks substantially and overall loss increases.
  • Gaussian Noise: At 10 dB, the Shannon Scaling Law achieves R2 0.9555 on Pythia, versus 0.8322 for the next-best Asymmetric law; on OLMo2 it achieves 0.8695.At 40 dB, it records 0.9895 for Pythia and 0.9830 for OLMo2.
  • Supervised Fine-Tuning: Across SFT datasets, the Shannon Scaling Law outperforms the strongest competitor with average R2 values of 0.936 versus 0.896 on GSM8K, 0.916 versus 0.875 on SiQA, and 0.937 versus 0.916 on StarCoder.Its advantage extends into loss basins, including GSM8K fits of 0.901 and 0.874 at learning rates 1e-4 and 2e-4.
  • Quantization: Under aggressive 2-bit quantization, baseline R2 falls to approximately 0.72 for Pythia and 0.61 for OLMo2, while the proposed method retains 0.9602 and 0.8869.The results indicate that the formulation captures non-monotonic saturation caused by reduced precision.
  • High-SNR Pretraining: On unperturbed pretraining trajectories, the law achieves R2 0.9889 on OLMo2 and 0.9915 on Pythia, treating standard monotonic scaling as a high-SNR special case.Across model families, it reports average R2 values of 0.9613 ± 0.03 for Pythia and 0.9585 ± 0.06 for OLMo2.

5 Beyond Fitting: Extrapolation and Analysis

The paper develops reduced and full Shannon laws for fitting and extrapolating noisy scaling behavior, finding that the full law is especially important when predicting across both model size and token count.

  • 5.1 Towards Parameter-Efficient Scaling Laws: 0.9541 vs. 0.9656 average R2 shows that the simplified law retains near-full fitting accuracy.At 10 dB, it achieves R2 = 0.9092 versus 0.8322 for the best baseline.
  • 5.2 Predictive Power via Extrapolation to Unseen Models and Tokens: 0.945 pooled R2 at j=15 lets Shannon-Simpl outperform QiD Law’s 0.862 and Law of Precision’s 0.743 in token extrapolation.The fits use early token checkpoints and predict the remaining checkpoints across six SNR levels.
  • 5.2 Predictive Power via Extrapolation to Unseen Models and Tokens: 0.837 pooled R2 at k=4 exceeds QiD Law’s 0.756, while OpenAI’s monotonic law collapses to −0.048 in model extrapolation.The models are fitted on the four smallest Pythia models and used to predict larger held-out models.
  • 5.2 Predictive Power via Extrapolation to Unseen Models and Tokens: 0.847 pooled R2 is achieved by the full Shannon Law when predicting an unseen 12B model and unseen tokens beyond a 180B-token fitting horizon.The headline setting uses the five smallest models, the first 12 checkpoints, and predicts token budgets from 180B to 307B.
  • 5.2 Predictive Power via Extrapolation to Unseen Models and Tokens: The full law is preferred for joint (N, D) extrapolation, where its 0.847 score exceeds Shannon-Simpl’s 0.673.Its additional parameters separately control signal scaling, interaction noise, and the irreducible noise floor.
  • 5.3 Fitted Exponents Reveal When Scaling Helps and When It Hurts: Under low SNR, model noise can grow faster than effective bandwidth, making larger models detrimental beyond a threshold.For GSM8K at LR = 1e−4, γ = 0.475 exceeds α = 0.321, aligning with loss-basin formation.
  • 5.3 Fitted Exponents Reveal When Scaling Helps and When It Hurts: Token noise exponents δ exceed signal exponents β across scenarios, including pretraining, so sufficiently large token budgets produce U-shaped degradation.The paper describes this degradation along the token axis as intrinsic rather than merely perturbation-induced.

6 Conclusion

The conclusion presents the Shannon Scaling Law as a noisy-channel framework for LLM scaling and reports that it captures degradation phenomena while extrapolating beyond its fitting range.

  • 6 Conclusion: The Shannon Scaling Law maps model parameters and training tokens to channel bandwidth and signal power in an information-transmission view of LLM training.The framework identifies a Shannon capacity for the model.
  • 6 Conclusion: Scaling without sufficient SNR leads to the performance degradation associated with catastrophic overtraining and quantization, which the law captures more faithfully than baselines.The conclusion frames this as a fundamental capacity effect.
  • 6 Conclusion: 0.847 pooled R2 is obtained when predicting the unseen 12B Pythia model up to 307B tokens from fits using models no larger than 6.9B and 180B tokens.The extrapolation extends 1.7× beyond the fitting range, while monotonic baselines collapse.

A.1 Implementation Details

The supplied implementation-details passages identify summary tables for model configurations, pretraining steps, and implementation details.

  • A.1 Implementation Details: Table 11 summarizes model configurations and pretraining steps.
  • A.1 Implementation Details: Table 12 contains implementation details.

A.2 Gaussian noise over sizes

Across Gaussian noise levels, the Shannon Law is compared with QiD and is described as fitting the transition from monotonic scaling to U-shaped degradation more robustly.

  • A.2 Gaussian noise over sizes: Figure 7 compares Shannon Law and QiD across Gaussian noise levels from 40 dB to 10 dB at fixed token budgets.The bottom curves represent high-SNR conditions and the top curves include noise-dominated settings such as 10 dB.
  • A.2 Gaussian noise over sizes: The Shannon Law tracks U-shaped degradation in noise-dominated settings where the QiD baseline systematically deviates.Both laws align in high-SNR regimes.
  • A.2 Gaussian noise over sizes: 0.87 and 0.95 average R2 scores exceed QiD’s 0.75 and 0.88 under insufficient training regimes.The Shannon Law tightly fits data across all noise levels simultaneously.

A.3 3D plots of Gaussian Noise Perturbations

The visualizations show that increasing perturbation strength transforms conventional monotonic scaling into U-shaped loss basins and, at extreme levels, catastrophic collapse. Across noisy regimes and tasks, the Shannon Law more faithfully fits the resulting loss surfaces than competing laws and remains robust across quantization schemes.

  • 3D surface alignment: At 10 dB, the Shannon surface achieves R2 = 0.954 versus 0.817 for the QiD law, which underestimates high-noise curvature.The Shannon surface tracks the data manifold more closely, while the QiD surface becomes rigid and detaches from it.
  • SFT perturbation landscapes: As SFT learning rate rises from 1e−5 to 1e−3, GSM8K, SiQA, and StarCoder landscapes transition from monotonic scaling to noise-dominated basins.At low learning rates, larger models and token budgets reduce loss; stronger optimization noise breaks this monotonic pattern.
  • U-shaped scaling: Loss basins appear at LR = 2e−4 for GSM8K, LR = 1e−4 for SiQA, and LR = 3e−4 for StarCoder, producing U-shaped behavior along both model-size and token axes.Scaling model size or training tokens initially lowers loss but eventually increases it beyond task-specific optima.
  • Catastrophic collapse: At the highest learning rate, the low-loss basin disappears and high loss spans both model-size and token dimensions across all three tasks.The passage attributes this collapse to the noise term overwhelming the capacity term.
  • Cross-task goodness of fit: Average R2 reaches 0.9252 on SiQA and 0.9055 on StarCoder for the Shannon Law, compared with 0.8948 and 0.8841 for the Asymmetric Law.These averages cover evaluations across varying SFT learning rates.
  • Quantization robustness: Under aggressive 2-bit quanto quantization, monotonic power laws collapse to R2 ≈0.01–0.03, while the Shannon Law preserves R2 = 0.9031.The same perturbation-aware advantage is reported across AWQ, bitsandbytes, and quanto evaluations.
Loading 2605.23901v1…