PaperlayerView exact revision on arXiv

Source-linked AI summary

Densing Law of LLMs

Chaojun Xiao, Jie Cai, Weilin Zhao, Guoyang Zeng, Biyuan Lin, Jie Zhou, Zhi Zheng, Xu Han, Zhiyuan Liu, Maosong Sun

arXiv:2412.04315v2cs.AIcs.CL

TL;DR

LLM scaling improves performance but creates growing efficiency challenges, motivating a metric that compares effectiveness across model sizes. The paper defines capability density using reference-model scaling and effective parameter size, then finds that open-source LLM density grows exponentially and doubles approximately every three months. The result suggests that comparable performance can increasingly be achieved with smaller models, within the evaluated scope.

  • Problem

    LLM scaling improves performance but creates major training and inference-efficiency challenges, especially for resource-constrained deployment.

    Source: §1 · p. 2

  • Method

    The paper defines capability density by fitting reference-model scaling relationships and comparing a target model’s effective parameter size with its actual parameter size.

    Source: §1 · p. 2, §2.2 · p. 5, §2.4 · p. 6

  • Results

    Capability density of open-source base LLMs increases exponentially over time and doubles approximately every three months on widely used benchmarks.

    Source: §3.3 · p. 8, §6 · p. 12

  • Takeaways & Limitations

    Within the evaluated scope, models with fewer parameters can increasingly achieve performance comparable to current state-of-the-art models.

    Source: §6 · p. 12

    Density measurements depend on benchmark quality and should incorporate more comprehensive and fair evaluations as LLM capabilities evolve.

    Source: §5 · p. 12

Abstract

from arXiv · show

Large Language Models (LLMs) have emerged as a milestone in artificial intelligence, and their performance can improve as the model size increases. However, this scaling brings great challenges to training and inference efficiency, particularly for deploying LLMs in resource-constrained environments, and the scaling trend is becoming increasingly unsustainable. This paper introduces the concept of ``\textit{capacity density}'' as a new metric to evaluate the quality of the LLMs across different scales and describes the trend of LLMs in terms of both effectiveness and efficiency. To calculate the capacity density of a given target LLM, we first introduce a set of reference models and develop a scaling law to predict the downstream performance of these reference models based on their parameter sizes. We then define the \textit{effective parameter size} of the target LLM as the parameter size required by a reference model to achieve equivalent performance, and formalize the capacity density as the ratio of the effective parameter size to the actual parameter size of the target LLM. Capacity density provides a unified framework for assessing both model effectiveness and efficiency. Our further analysis of recent open-source base LLMs reveals an empirical law (the densing law)that the capacity density of LLMs grows exponentially over time. More specifically, using some widely used benchmarks for evaluation, the capacity density of LLMs doubles approximately every three months. The law provides new perspectives to guide future LLM development, emphasizing the importance of improving capacity density to achieve optimal results with minimal computational overhead.

Highlights

The paper introduces capability density to evaluate LLM quality across scales by combining effectiveness and efficiency. It reports an exponential increase in maximum capability density, with doubling approximately every 3.3 months under the evaluated benchmarks and reference models.

Highlights

  • Capability density evaluates LLM quality across scales while considering both effectiveness and efficiency.It is defined as effective parameter size divided by actual parameter size.Source: Highlights · p. 1
  • Maximum capability density exhibits exponential growth over time, forming the paper’s densing law.The relationship is modeled as ln(ρmax) = At + B.Source: Highlights · p. 1
  • A≈0.007 with R2≈0.93, and maximum capability density doubles approximately every 3.3 months across five widely used benchmarks.Figure 1 measures popular open-source LLMs using performance on five benchmarks.Source: Highlights · p. 1, Figure 1 · p. 1
  • Around three months of progress can enable a model with half the parameter size to achieve performance comparable to current state-of-the-art LLMs.This implication follows the reported density-doubling trend.Source: Highlights · p. 1
  • The reported density growth rate depends on the specific evaluation benchmarks and reference models.This qualifies the empirical densing trend.Source: Highlights · p. 1

1 Introduction

The paper introduces capability density to compare LLM effectiveness and efficiency across scales, then analyzes open-source models to identify an exponential density-growth trend and its implications.

1 Introduction

  • Increasing LLM scale improves performance but creates inference-cost and deployment pressures, especially for resource-constrained devices.These pressures motivate a metric and law that address effectiveness and efficiency together.Source: §1 · p. 2
  • Capability density is the ratio of effective parameter size to actual parameter size, using a reference model to estimate equivalent performance requirements.This metric provides a unified view of LLM effectiveness and efficiency across different scales.Source: §1 · pp. 2–3

1.1 Key Findings

  • The study analyzes 29 open-source pretrained base models and derives implications from their observed density trend.These implications include decreasing inference costs for equivalent downstream performance and faster growth of effective parameter capacity on similarly sized chips.Source: §1.1 · p. 3
  • The Densing Law states that maximum capability density grows exponentially over time, with ln(ρmax) = A · t + B.Here, ρmax denotes the maximum capability density at time t.Source: §1.1 · p. 3
  • A ≈0.007 implies that maximum density doubles approximately every three months across five widely used benchmarks.The estimate may vary slightly with evaluation benchmarks and reference models.Source: §1.1 · p. 3
  • After ChatGPT’s release, the growth rate of LLM density increased by 50%.The comparison examines maximum-density trends before and after ChatGPT’s release.Source: §1.1 · p. 3
  • Pruning and distillation usually produce smaller models with lower density than their original models, so compression alone does not ensure density improvement.The paper therefore argues for compression methods that improve the density of smaller models.Source: §1.1 · p. 4
  • The paper recommends shifting LLM development from performance-centric scaling toward density-centric training to achieve strong results with lower computational costs.It frames density as a trade-off between effectiveness and efficiency and connects this goal to more sustainable scaling.Source: §1.1 · p. 4

2 Density for Large Language Models

This section defines LLM density by comparing a model’s actual parameter count with the reference-model size needed to match its downstream performance. It estimates that equivalent size through loss and performance scaling functions, then uses the resulting density to compare model quality across scales.

2.1 Overall Framework and Definition

  • The framework fits a reference-model scaling function and inverts it to estimate the parameter count required to match a target model’s downstream performance.The procedure first relates parameter size to performance and then computes effective parameter size through the inverse function.Source: §2.1 · pp. 4–5

2.2 Loss Estimation

  • Conditional loss is estimated as a power-law function of parameter size and training-token count, focusing on answer generation given task instructions.For downstream tasks, the loss includes task-specific outputs such as reasoning steps and final answers; GPT-4o generates reasoning steps when datasets lack them.Source: §2.2 · p. 5

2.3 Performance Estimation

  • A sigmoid maps estimated loss to bounded downstream performance because limited-compute scaling-law models often remain near random guessing.The mapping approaches random-guessing performance at large loss and an upper bound at small loss.Source: §2.3 · pp. 5–6

2 Density for Large Language Models

  • Density is defined as the ratio of effective parameter size to actual parameter size, indicating how efficiently a model converts parameters into performance.The effective size is inferred for a reference model trained with D0 = 1T tokens.Source: §2.4 · p. 6

2.4 Density

  • After fitting loss and performance equations, the method fixes training data at D0 = 1T tokens and computes effective parameter size before dividing by actual size.Higher density corresponds to better performance at the same parameter scale, motivating density improvements under deployment resource constraints.Source: §2.4 · p. 6

3 Density Evolution

The paper estimates capability density through reference-model scaling curves and finds that LLM density has risen exponentially, with implications for inference cost, compression, and deployment.

3.1 Evaluation Settings

  • Capability density is estimated by fitting loss and performance scaling curves from reference models, then comparing each model’s effective and actual parameter sizes.The evaluation uses loss-estimation models, well-trained performance-estimation models, and downstream benchmarks.Source: §3.1 · pp. 6–7

3.2 Loss and Performance Estimation Results

  • The two-step estimation process fits downstream performance as a function of loss and successfully predicts larger models using smaller models for curve fitting.Models with fewer than 4 billion parameters fit the loss-performance curve, while models with tens of billions of parameters are reserved for prediction.Source: §3.2 · p. 8

3.3 Densing Law

  • Density increased from below 0.1 for Llama-1 to about 3 for Gemma-2-9B and MiniCPM-3-4B, while larger parameter counts did not always yield higher density.The paper attributes the trend partly to larger and higher-quality pre-training data and notes that extremely large models may be less cost-effective.Source: §3.3 · p. 8
  • A ≈0.0073 and R2 = 0.912 imply that maximum capability density doubles approximately every 95 days.The paper models logarithmic maximum density as linear in time.Source: §3.3 · pp. 8–9

3.4 Corollaries of Densing Law

  • Under equal performance, exponential density growth implies exponentially declining parameter counts and inference costs, with API pricing falling 266.7 times from GPT-3.5 to Gemini-1.5-Flash.The reported prices were $20 and $0.075 per million tokens, respectively, and inference costs roughly halved every 2.6 months.Source: §3.4 · p. 9
  • The combined growth of model density and chip transistor density implies that the largest effective model supported at a fixed chip price doubles approximately every 88 days.The paper links this trend to potentially more capable models running on consumer-grade devices.Source: §3.4 · pp. 9–10
  • ChatGPT’s release coincided with a 50% faster density-growth rate, increasing the fitted slope from A ≈0.0048 to A ≈0.0073.The paper associates the acceleration with increased investment and more accessible high-quality open-source models.Source: §3.4 · p. 10, Figure 4 · p. 10
  • Compression did not consistently improve density: only Gemma-2-9B exceeded its original model, while other compressed models had lower density.The paper suggests insufficient training of smaller compressed models as a possible explanation.Source: §3.4 · pp. 10–11, Figure 5 · p. 11

4 Discussion

The discussion frames capability density as a relative metric whose validity depends on benchmark coverage and data contamination, while treating the law’s duration as uncertain.

Accurate Capability Measurement

  • Capability density is relative rather than an absolute measure of intelligence because current technology cannot accurately quantify absolute LLM capability.The estimate may be biased by the limited benchmark set and potential test-data contamination.Source: §4 · p. 11

Connection between Densing Law and Scaling Law

  • The paper connects Densing Law to Scaling Law by proposing that improvements in data, algorithms, and architecture can increase α and reduce loss at a fixed parameter size.The stated Scaling Law is L = AN^-α.Source: §4 · p. 11

Period of Validity of Densing Law

  • The duration of exponential density growth is uncertain because evaluation datasets must be updated as models approach saturation on existing benchmarks.The authors expect the law to remain effective for a considerable period but do not establish a fixed validity horizon.Source: §4 · p. 12

5 Limitations and Future Directions

The proposed capability-density evaluation is limited by benchmark coverage and fairness, and currently applies only to language models rather than multimodal systems. Future directions include broader datasets, anti-overfitting evaluation, multimodal density measures, and inference-FLOP-based assessment.

Fair and Comprehensive Evaluation

  • Benchmark quality strongly affects capability-density measurements, making comprehensive and fair evaluation an ongoing limitation.Existing benchmarks may not capture expanding abilities such as complex reasoning, while benchmark overoptimization can inflate scores.Source: §5 · p. 12

Multi-modal Density

  • Capability-density measurement currently focuses on language models, leaving multimodal density and its trends for future work.The paper identifies multimodal density evaluation as an important research direction as multimodal applications increase.Source: §5 · p. 12

Inference Densing Law

  • Parameter size may be insufficient for reasoning-focused evaluation, motivating a future density measure based on effective versus actual inference FLOPs.The proposed shift reflects the role of additional inference computation in enabling deeper reasoning on complex tasks.Source: §5 · p. 12

6 Conclusion

The paper introduces capability density to quantify LLM training quality and efficiency, then analyzes open-source base models released since 2023. It finds that capability density increases exponentially over time and doubles approximately every three months, indicating rapid gains in LLM efficiency.

6 Conclusion

  • Capability density measures LLM training quality and efficiency, providing a method to quantify the recent trend toward efficient models.The paper applies this method to open-source base LLMs released since 2023.Source: §6 · p. 12
  • Capability density increases exponentially over time, with evaluation results indicating that density doubles approximately every three months.Within three months, a model with half the parameters can achieve performance comparable to current state-of-the-art models.Source: §6 · p. 12
  • The finding highlights the rapid development and increasing efficiency of LLMs while motivating continued improvement of capability density with minimal computational costs.The paper presents this law and its corollaries as perspectives for future LLM development.Source: §6 · p. 12
Loading 2412.04315v2…