Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs

1 INTRODUCTION

• Existing scaling laws omit inference costs, a dominant expense in deployment, and the accuracy-efficiency trade-off remains an open question.Prior approaches either require estimating lifetime generated tokens or limit architectural conditioning to aspect ratio, leaving broader factors such as hidden size and GQA unaddressed.Source: §1 · p. 2
• The study fixes layer count while investigating how hidden size, GQA, and the MLP-to-attention ratio affect inference efficiency and accuracy.It compares models with identical parameter counts and trains over 200 models from 80M to 297M parameters on up to 30B tokens.Source: §1 · p. 2
• A conditional extension of Chinchilla scaling laws incorporates architectural parameters to identify designs balancing inference efficiency and performance.The framework is guided by empirical comparisons of architectural effects on accuracy and inference efficiency.Source: §1 · p. 2
• Although larger models generally have lower inference throughput, Qwen2.5-1.5B outperforms smaller Qwen3-0.6B, with differences in hidden size, GQA, and MLP-to-attention ratio.The models have the same number of layers.Source: Figure 2 · p. 2
• Up to 42% higher inference throughput than the LLaMA-3.2-3B architecture, while maintaining better accuracy, is achieved by the derived optimal 3B architecture under identical training setups.The framework was validated by fitting scaling laws on models between 80M and 297M parameters and evaluating predictions when scaling to 3B parameters.Source: §1 · pp. 2–3

2 BACKGROUND

• Chinchilla models loss as a power-law function of parameter count N and training-token count D, with learnable parameters A, B, E, α, and β.The law is used to predict model performance during scaling.Source: §2 · p. 3
• The scaling-law fit combines runs varying training tokens at fixed parameter count with runs varying both parameters and tokens under a fixed compute budget.The combined data are used to fit the law's learnable parameters.Source: §2 · p. 3
• Chinchilla allocates compute between parameters N and training tokens D under resource constraint C, whereas this paper searches architectures at fixed parameter and token budgets.The paper does not address optimal compute allocation between model size and training data.Source: §2 · p. 3

3.1 MODEL ARCHITECTURE VARIATIONS

• The search fixes the layer count and varies hidden size and MLP-to-attention ratio; the number of layers still varies across parameter-budget levels.The architecture study also accommodates GQA, with attention-head choices adjusted to support architectural variants.Source: §3.1 · pp. 3–4

3.2 INFERENCE EFFICIENCY

• Larger hidden sizes, higher MLP-to-attention ratios, and increased GQA improve inference throughput in the tested variants.The authors attribute gains partly to reduced FLOPs and a smaller KV cache, which lowers inference I/O cost.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 A CONDITIONAL SCALING LAW

• Training loss has a U-shaped relationship with normalized hidden size, with nearly identical optima across model sizes; excessively large hidden sizes can degrade accuracy.The cited analysis links oversized hidden sizes to fewer attention heads.Source: §3.3 · p. 5
• Training loss also has an interior optimum over the MLP-to-attention ratio, and moving either direction away from it degrades performance.The U-shaped relationship is observed while other architectural factors are held fixed.Source: §3.3 · p. 5, Figure 5 · p. 5
• The conditional law first obtains Chinchilla-optimal loss for given parameter and token counts, then calibrates architecture-specific loss relative to that reference.Its functional forms treat hidden size and MLP-to-attention ratio effects as separable and model each with c0 + c1 log x + c2/x.Source: §3.3 · pp. 5–6

3.4 SEARCHING FOR INFERENCE-EFFICIENT ACCURATE MODELS

• Architecture search maximizes inference efficiency subject to a loss threshold, then locally enumerates feasible GQA values because GQA has no consistent continuous loss relationship.The search can stop when performance falls below the GQA=4 baseline.Source: §3.4 · p. 6

Training Setup

• Models from 80M to 3B non-embedding parameters vary hidden size and MLP-to-attention ratio, with other factors held fixed, and train on 100 tokens per parameter.Training data are sampled from Dolma-v1.7; the token budget is five times Chinchilla-optimal.Source: §4 · pp. 6–7

Predictive Accuracy

• Across the scale-transfer tasks, predictions show low MSE and high Spearman correlation against actual pretraining loss.MSE measures loss prediction error, while Spearman correlation compares predicted and actual rankings.Source: Figure 6 · p. 7, §4 · p. 7

Inference and LLM Evaluation Setup

• Inference throughput is averaged over five vLLM runs, while pretrained models are evaluated zero-shot on nine benchmarks.The default inference setup uses 4096 input tokens and 1024 output tokens on A100 GPUs.Source: §4 · p. 7

Fitting Scaling Laws

• The conditional law is fitted by Levenberg-Marquardt, using empirically searched optimal losses below 1B, and tested progressively from 80M through 1B.The sequence fits 80M then evaluates 145M; fits 80M and 145M then evaluates 297M; and fits through 297M then evaluates 1B.Source: §4 · p. 7

5.1 OPTIMAL MODEL ARCHITECTURE

• Up to 42% higher inference throughput is achieved by Surefire-1B and Surefire-3B than by LLaMA-3.2-1B and LLaMA-3.2-3B, while Surefire also outperforms them downstream.Figure 7 reports Surefire as consistently efficient across batch sizes.Source: Figure 7 · p. 8, §5.1 · p. 9

5 EXPERIMENT RESULTS

• Across Tasks 1–3, conditional scaling laws achieve low MSE and high Spearman correlation when predicting losses at different model scales.Fits use 80M data to evaluate on 145M, 80M and 145M to evaluate on 297M, and 80M, 145M and 297M to evaluate on 1B.Source: §5 · p. 8
• Including mlp-to-attention ratios outside [0.5, 5] clearly degrades Task 3 Spearman correlation, supporting exclusion of extreme outliers from fitting.The tested outliers include ratios such as 0.1 and 12.6.Source: §5 · p. 8
• Additive and multiplicative calibration achieve similar Task 3 MSE and Spearman correlations, and more complex joint formulations do not improve predictive performance.Both simple calibrations assume separable effects of mlp-to-attention ratio and hidden size on loss.Source: §5 · p. 8

5.1 OPTIMAL MODEL ARCHITECTURE

• Panda-3B◦, fitted using only 1B data, has lower training loss and comparable downstream accuracy to Panda-3B in the 3B ablation.The comparison also includes LLaMA-3.2-3B; the authors suggest fitting at roughly one third of the target scale can be sufficient or preferable.Source: Table 2 · p. 9, §5.1 · pp. 9–10
• Panda-1B exceeds LLaMA-3.2-1B by 2.1% on average across downstream tasks and has the lowest training loss among the exhaustively trained 1B variants.Both models were trained on 100B tokens under the same setup.Source: Figure 7 · p. 8, §5.1 · p. 9
• Panda-3B exceeds the open-weight LLaMA-3.2-3B configuration by 0.6% on average across downstream tasks after training on 100B tokens.Source: §5.1 · p. 9
• For 3B loss prediction, fitting with only 1B variants gives lower MSE and higher Spearman correlation than the progressive fitting-data strategy.The authors refit with multiplicative calibration using only 1B variants, suggesting coefficients shift with model size.Source: §5.1 · p. 9, Figure 8 · p. 10

• Existing scaling laws characterize performance across model size, dataset size and quality, and compute, while recent MoE studies incorporate expert counts, granularity, active parameters, and sparsity.Source: §6 · p. 10

• The study does not evaluate 7B models, establish scaling laws for MoE architectures, or assess how results change under post-training.Its analysis is restricted to dense models and pretraining.Source: §7 · p. 10

• The framework trains small models to fit a conditional scaling law, solves for a predicted optimal architecture, and locally searches over GQA to maximize inference efficiency.Source: §8 · p. 10
• Models up to 3B parameters achieve up to 42% higher inference throughput and 2.1% accuracy gains across nine downstream tasks.Source: §8 · p. 10

REPRODUCIBILITY STATEMENT

• Training used Megatron-LM, inference used vLLM and SGLang, and standardized evaluations used lm-eval-harness.The authors state that all experiments used publicly available frameworks.Source: REPRODUCIBILITY STATEMENT · p. 11

A LLM USAGE

• An LLM was used to correct grammar in the draft, not to generate research ideas.Source: A LLM USAGE · p. 16

B ADDITIONAL RELATED WORK

• Related work covers Transformers and LLMs, serving systems, and inference-efficient model design.The discussion names examples across each area, including vLLM and SGLang for serving systems.Source: B ADDITIONAL RELATED WORK · p. 16

C OPEN-WEIGHTED MODEL ARCHITECTURES

• Table 3 overviews the open-weight model architectures used in the paper, including layers, hidden size, attention heads, and intermediate size.Source: C OPEN-WEIGHTED MODEL ARCHITECTURES · p. 16, Table 3 · p. 16

D MODEL ARCHITECTURES

• Table 4 lists architectures of models trained in the paper, including non-embedding parameter count and MLP-to-attention ratio.It also reports layer count, hidden size, attention heads, and intermediate size.Source: Table 4 · p. 16, D MODEL ARCHITECTURES · p. 16

E HYPER-PARAMETERS

• Table 5 lists the hyperparameters used for training.Source: Table 5 · p. 20, E HYPER-PARAMETERS · p. 20

F ADDITIONAL INFERENCE EVALUATION RESULTS OVER A100 GPUS

• At fixed parameter budgets, larger hidden sizes and MLP-to-attention ratios yield higher throughput, while increasing GQA also raises throughput in the reported A100 evaluations.The evaluations span multiple model scales and batch sizes.Source: Figure 9 · p. 21, Figure 10 · p. 21, F ADDITIONAL INFERENCE EVALUATION… · p. 21
• Surefire-1B and Surefire-3B consistently achieve higher inference efficiency than their LLaMA-3.2 counterparts across batch sizes on A100 and H200 GPUs.The comparisons use SGLang with 4096 input and 1024 output tokens.Source: Figure 15 · p. 23, Figure 19 · p. 24

G ADDITIONAL INFERENCE EVALUATION RESULTS OVER H200 GPUS

• On H200 GPUs, Qwen3 variants show higher throughput with larger hidden sizes, MLP-to-attention ratios, and GQA across model scales and batch sizes.The reported results include vLLM and SGLang evaluations.Source: Figure 16 · p. 23, Figure 17 · p. 24, Figure 18 · p. 24

H DETAILED THROUGHPUT STATISTICS

• At 3B, LLaMA-3.2-3B-HF achieves good accuracy but is not inference-efficient, whereas Qwen2.5-3B-HF is inference-efficient but less accurate.OLMo-2-1B-HF is relatively close to the predicted optimal MLP-to-attention ratio but remains inference-inefficient due to hidden dimension and GQA choices.Source: H DETAILED THROUGHPUT STATISTICS · p. 27

I ADDITIONAL RESULTS: LOSS VS. GQA

• With parameter count, hidden size, and MLP-to-attention ratio fixed, training loss varies substantially as GQA changes.The reported loss–GQA relationship is highly fluctuating across model sizes.Source: I ADDITIONAL RESULTS: LOSS VS. GQA · p. 28, Figure 24 · p. 28
• At fixed total heads, increasing query heads expands query projection dimensionality but reduces KV-head count and KV expressivity; decreasing query heads reverses these trade-offs while reducing query and KV projection dimensionality.These opposing effects make the relationship between GQA and training loss non-smooth.Source: I ADDITIONAL RESULTS: LOSS VS. GQA · p. 28
• Because GQA settings vary across related model sizes and lack a monotonic or predictive loss theory, the paper treats GQA as a model-specific hyperparameter for local search rather than part of the continuous scaling law.Source: I ADDITIONAL RESULTS: LOSS VS. GQA · p. 28

J MORE ABLATION STUDY

• Outlier data points harm the scaling-law fit, while multiplicative and additive calibrations have nearly identical MSE and Spearman values; joint and non-separable calibrations perform worse than multiplicative calibration.The outliers are models with MLP-to-attention ratios below 0.5 or above 5.Source: Figure 25 · p. 29, Figure 26 · p. 29, J MORE ABLATION STUDY · p. 29

K INFERENCE FLOPS ANALYSIS

• Increasing hidden size or the MLP-to-attention ratio reduces the 2Tdq term, lowering total inference FLOPs.The analysis focuses on non-embedding FLOPs.Source: K INFERENCE FLOPS ANALYSIS · p. 30

L MORE LARGE-SCALE TRAINING RESULTS

• Tables 9 and 10 report detailed downstream-task results for 1B and 3B models, respectively, covering nine tasks at each scale.Source: L MORE LARGE-SCALE TRAINING RESULTS · p. 31, Table 9 · p. 31, Table 10 · p. 31

M MOE INFERENCE

• For MoE models, larger hidden sizes and higher Active-Experts-to-Attention ratios improve inference throughput, with total active parameters fixed, GQA set to 4, and batch size 2048 in the reported study.Source: M MOE INFERENCE · p. 32, Figure 27 · p. 32