PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Inference-Efficient Language Models

Song Bian, Minghao Yan, Shivaram Venkataraman

arXiv:2501.18107v2cs.LGcs.AIcs.CL

TL;DR

Existing scaling laws omit inference costs, although those costs matter in real-world deployment. This work extends scaling-law-guided model selection to account for architecture and inference latency, and reports that Morph-1B is faster while maintaining downstream accuracy. The authors also identify a 7B-model scaling boundary due to resource limitations.

  • Problem

    Existing scaling laws balance parameters and training tokens but omit inference costs, which can dominate long-term deployment expenses.

    Source: §1 · pp. 1–2

  • Method

    The authors extend Chinchilla scaling laws to incorporate architecture and inference-latency constraints, then use predicted losses to rank architectural choices.

    Source: §1 · p. 2, §2.2 · p. 4

  • Results

    Morph-1B improves inference latency by 1.8× while maintaining downstream-task accuracy compared with open-source models of similar size.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The inference-efficient scaling law can capture accuracy-efficiency tradeoffs for architectural optimizations such as GQA and MLA.

    Source: §1 · p. 2

    Resource limitations prevented scaling training to include 7B models.

    Source: §6 · p. 9

Abstract

from arXiv · show

Scaling laws are powerful tools to predict the performance of large language models. However, current scaling laws fall short of accounting for inference costs. In this work, we first show that model architecture affects inference latency, where models of the same size can have up to 3.5x difference in latency. To tackle this challenge, we modify the Chinchilla scaling laws to co-optimize the model parameter count, the number of training tokens, and the model architecture. Due to the reason that models of similar training loss exhibit gaps in downstream evaluation, we also propose a novel method to train inference-efficient models based on the revised scaling laws. We perform extensive empirical studies to fit and evaluate our inference-aware scaling laws. We vary model parameters from 80M to 1B, training tokens from 1.6B to 30B, and model shapes, training 63 models. Guided by our inference-efficient scaling law and model selection method, we release the Morph-1B model, which improves inference latency by 1.8x while maintaining accuracy on downstream tasks compared to open-source models, pushing the Pareto frontier of accuracy-latency tradeoff. Notably, our experiments reveal that wider and shallower models can yield efficiency gains while preserving accuracy.

1. Introduction

Existing scaling laws do not account for inference costs, despite their importance in deployment. This work incorporates architecture and inference latency into scaling-law-guided model selection, reporting a faster Morph-1B with maintained downstream accuracy.

1. Introduction

  • Morph-1B improves inference latency by 1.8× while maintaining downstream-task accuracy compared with open-source models of similar size.Figure 1 summarizes the comparison.Source: Figure 1 · p. 1, §1 · p. 2
  • Existing scaling laws omit inference costs, which can dominate long-term use; practical choices also depend on training data size and deployment-device memory.The authors argue that scaling laws should account for data size, device memory, and inference latency.Source: §1 · pp. 1–2
  • The paper extends Chinchilla scaling laws to consider architecture and develops a ranking method because similar training losses can correspond to different downstream-task accuracy.The method uses predicted-loss rankings to compare architectural choices.Source: §1 · p. 2
  • The fitting study trains more than 60 models from 80M to 339M parameters for up to 13B tokens, then evaluates predictive power on larger models.The larger evaluation models exceed 1B parameters and use 20B tokens.Source: §1 · p. 2
  • The inference-efficient scaling law is more accurate and robust than Chinchilla, and predicts scaled-up model loss using 6 data points and 85 A100 GPU hours.The authors report that overtraining data is important for obtaining an accurate scaling law.Source: §1 · p. 2

2. Scaling Laws

The section moves from compute-constrained loss prediction to scaling laws that include model shape and inference latency. It also proposes ranking candidate architectures by latency and predicted loss, then selecting among trained models using downstream-task performance.

2.1. Preliminaries

  • Chinchilla scaling laws predict training loss from model size and training tokens, with D ≈ 20N as the compute-optimal solution under a fixed FLOPs budget.Here, N is the parameter count and D is the number of training tokens.Source: §2.1 · p. 3

2.2. Inference-Efficient Scaling Laws

  • Existing scaling laws do not reflect common overtraining practices or account for inference latency, which can make smaller models slower than larger ones.The section notes that LLaMA-3-8B and Gemma-2-9B use 44–93× more tokens than the Chinchilla recommendation, and that MiniCPM-1B has higher latency than Qwen2.5-14B.Source: §2.2 · p. 3
  • At a fixed parameter count, wider, shallower models have lower inference latency and consistently higher throughput; latency also rises linearly with layer count at fixed hidden size.The latency and throughput evaluations vary architecture while controlling parameter count, and the throughput pattern holds across batch sizes.Source: Figure 3 · p. 3, §2.2 · p. 4, Figure 4 · p. 4
  • The revised loss law multiplies the Chinchilla loss expression by (1 + εR^γ), where R = dmodel/nlayers, to incorporate model shape.Its learned parameters are A, B, E, α, β, γ, and ε; the reported predictions capture training loss across different training durations.Source: Figure 5 · p. 4, §2.2 · p. 5

2.3. Methodology

  • Because similar training losses do not robustly predict individual downstream-task accuracies, the method ranks candidates by measured latency and predicted loss, then evaluates top-k trained models on downstream tasks.The user chooses k, and the release selection considers both inference latency and downstream-task performance.Source: §2.3 · p. 5, Figure 7 · p. 5, Figure 6 · p. 5

3. Experiments

The experiments train and evaluate decoder-only language models, then fit inference-efficient scaling laws to loss measurements across model sizes, shapes, and training-token counts.

3.1. Experimental Setup

  • Models use a GPT-2/LLaMA-like decoder-only architecture, GPT-NeoX tokenization, and one-epoch training on sampled DCLM-Baseline data.The setup trains models up to 1.5B parameters and 30B tokens.Source: §3.1 · p. 6
  • Inference latency is measured on one 40GB A100 with 128 input and 256 output tokens; downstream accuracy uses zero-shot evaluation.The evaluated tasks include ARC, BoolQ, HellaSwag, PIQA, MMLU, and others.Source: §3.1 · p. 6
  • The study evaluates loss predictions with relative error, MSE, and R2, and compares predicted and actual model rankings with Spearman correlation.The passages define actual loss as ψ and predicted loss as ψ̂.Source: §3.1 · p. 6

3.2. Fitting Scaling Laws

  • The authors fit Eq. (4) with Levenberg–Marquardt least-squares curve fitting.The fitting dataset spans model sizes, shapes, and training-token amounts.Source: §3.2 · p. 6
  • 63 models spanning 80M–339M parameters and 1.6B–12.8B training tokens provide data to fit and evaluate the scaling laws.The fitting procedure sets α, β, and γ equal to simplify the fit.Source: §3.2 · p. 6

4. Results

The results evaluate inference-efficient scaling-law predictions, use predicted loss and latency to select model shapes, and test robustness under fitting-data changes. Morph-1B-v1 is reported to reduce inference latency without losing accuracy relative to Open-LM-1B.

4.1. Prediction acccuracy

  • MSE falls from 0.0033 to 0.0006 and R2 rises from 0.9895 to 0.9982 for the inference-efficient law versus Chinchilla.Figure 8 compares predicted and actual loss; the inference-efficient law's relative prediction error is below 1.2%, versus 2.7%–4.1% for Chinchilla.Source: Figure 8 · p. 7, §4.1 · p. 7
  • For 1B-model loss ranking, Spearman correlation is 1.00 for the inference-efficient law versus -0.40 for Chinchilla.The authors state that ranking predicted losses matters more than their absolute values for their model-selection method.Source: §4.1 · p. 7, Figure 8 · p. 7

4.2. Inference-Efficient Models

  • The selection procedure varies Open-LM-1B's aspect ratio, measures latency on one A100, then trains three candidates chosen using latency and predicted loss.The selected models are evaluated on 20 downstream tasks.Source: §4.2 · p. 7
  • Morph-1B-v1 has 1.8× lower inference latency than Morph-1B, without any loss in accuracy, as illustrated in Table 2.Source: §4.2 · pp. 7–8

4.3. Insights from Scaling Laws Fitting

  • With only Chinchilla-optimal fitting data, the inference-efficient law has MSE 0.1165 and R2 0.6293, versus Chinchilla's 0.9825 and -2.1259.Figure 9 compares the laws when over-training data are excluded.Source: §4.3 · p. 8, Figure 9 · p. 8
  • Both laws' performance deteriorates when predicting losses for over-trained models, so over-training data are essential to fit the inference-aware law.Source: §4.3 · p. 8
  • With randomly selected model shapes, the inference-efficient law has MSE 0.0008 and R2 0.9973, versus Chinchilla's 0.0198 and 0.9369.Its 1B-model relative prediction error is below 0.72%, compared with 11.8%–13.4% for Chinchilla.Source: §4.3 · p. 8, Figure 10 · p. 8
  • Fitting with six data points reduces the reported fitting cost from 450 to 85 A100 GPU hours.Source: §4.3 · p. 9

5. Related Work

Related work covers scaling laws for predicting language-model behavior, systems for improving inference serving, and model-design approaches to compute efficiency.

Related Work

6. Limitations and Future Work

The authors identify limits to the study's scale and note that inference systems and attention architectures may affect efficiency trade-offs.

Limitations and Future Work

  • Resource limits prevented scaling training to 7B models; newer inference systems and MQA, GQA, or MLA may alter efficiency trade-offs.The authors say their approach can quantify and predict how architectural optimizations affect accuracy-efficiency trade-offs.Source: §6 · p. 9

7. Conclusion

The paper develops inference-efficient scaling laws and a model-selection method, then uses them to design Morph-1B, which improves inference efficiency while maintaining downstream accuracy against similar-sized open-source models.

7. Conclusion

  • Morph-1B improves inference efficiency while maintaining downstream-task accuracy compared with similar-sized open-source models.The model was designed using the inference-efficient scaling law.Source: §7 · p. 9
  • The paper proposes inference-efficient scaling laws to jointly optimize inference cost and model loss.It also develops a method to train and rank inference-efficient models using the scaling law.Source: §7 · p. 9
  • An empirical study develops scaling laws to guide inference-efficient model-architecture design.Source: §7 · p. 9

Impact Statement

The work aims to train more inference-efficient language models, potentially reducing deployment costs and associated environmental impacts.

Impact Statement

A. Hyperparameters and Model Architectures

The experiments use specified hyperparameters and bfloat16 with AdamW, count parameters from model parameters, and vary configurations to study architecture effects on loss and inference.

A. Hyperparameters and Model Architectures

B. Results over A30 GPUs

On A30 GPUs, inference latency depends on both parameter count and architecture; latency increases linearly with layer count, and this pattern persists under a different input-output token setting.

B. Results over A30 GPUs

  • The importance of parameter count and architecture for inference efficiency is also observed across evaluated open-source LLMs.Figure 11 covers LLaMA, Qwen, Gemma, and MiniCPM using Hugging Face generate on a single NVIDIA A30 GPU, with 128 input and 256 output tokens by default.Source: B. Results over A30 GPUs · p. 15, Figure 11 · p. 15
  • The shape-related latency findings remain valid with 1,024 input tokens and 128 output tokens.Figure 13 tests the layer-count, hidden-size, and aspect-ratio relationships under this token setting.Source: B. Results over A30 GPUs · p. 15, Figure 13 · p. 16
  • Inference latency increases linearly with layer count, and adjusting model configurations can reduce latency.Figure 12 examines layer count at constant hidden size; its other panels relate latency to hidden size and aspect ratio.Source: Figure 12 · p. 15, B. Results over A30 GPUs · p. 15

C. More Results over A100 GPUs

At a fixed parameter count, wider and shallower models consistently achieve lower Time To First Token (TTFT) on an A100 GPU.

C. More Results over A100 GPUs

D. Results over vLLM

On vLLM, inference efficiency depends on architecture as well as parameter count, and latency increases linearly with layer count. Further experiments report that inference-efficient scaling laws are more robust than Chinchilla scaling laws across random model-shape selections.

D. Results over vLLM

  • Inference efficiency over vLLM depends on architecture as well as parameter count, and latency increases linearly with layer count.The evaluations use an A100 GPU; the accompanying figures describe latency by model, layer count, hidden size, and aspect ratio.Source: D. Results over vLLM · p. 17, Figure 16 · p. 17, Figure 15 · p. 17

E. More Scaling Laws Fits

F. Evaluation Dataset Details

G. Open-Source Model Configurations

H. Parameter Fits

I. Contribution Statement

  • Song developed the inference-efficient scaling laws and model-training methodology and conducted the experiments; Minghao proposed investigating configuration effects, while Shivaram recommended training a more inference-efficient 1B model that maintains downstream accuracy.Source: §I · p. 21
Loading 2501.18107v2…