PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling of Capability and Efficiency at Inference Time in Large Reasoning Models

Moritz Laber, Zohair Shafi, Germans Savcisens, Brennan Klein, Matteo Chinazzi, Samuel V. Scarpino, Albert-László Barabási, Alessandro Vespignani, Tina Eliassi-Rad

arXiv:2609.27166v2cs.LGphysics.soc-ph

TL;DR

The paper asks how model size affects capability and efficiency in reasoning, where inference-time compute adds a cost for each query. Using hierarchical Bayesian models to assess DeepSeek-R1-Distill models, it finds capability gains with diminishing returns and little evidence of systematic efficiency gains.

  • Problem

    How capability and efficiency depend jointly on problem hardness and model size remains poorly understood, despite inference-time compute carrying a marginal cost for each query.

    Source: §I · pp. 1–2

  • Method

    Hierarchical Bayesian models infer capability and efficiency from correct-solution counts and output lengths for DeepSeek-R1-Distill models.

    Source: A. Evaluation Strategy · pp. 2–3, §I · p. 2

  • Results

    Larger models are more capable but show diminishing capability gains, while correct responses provide little evidence of systematic efficiency improvement.

    Source: §I · p. 2, §III · p. 7

  • Takeaways & Limitations

    In the studied setting, scaling model size improves capability with diminishing returns, while efficiency shows little to no improvement.

    Source: §III · p. 7

    Whether these findings generalize beyond the DeepSeek-R1-Distill family and four reasoning tasks remains an open question.

    Source: §III · p. 8

Abstract

from arXiv · show

Capability and efficiency are two key dimensions of reasoning in large language models (LLMs). Capability refers to the ability to solve a given problem correctly, whereas efficiency refers to the ability to do so with limited resources. When LLMs use Chain-of-Thought (CoT) reasoning to solve problems of controlled hardness, both the number of problems solved correctly and the number of tokens required to reach a correct answer depend on problem hardness and model size. However, how these factors jointly shape capability and efficiency remains poorly understood. Here, we use hierarchical Bayesian models to evaluate the capability and efficiency of LLMs from the DeepSeek-R1-Distill model family across four classes of arithmetic and algorithmic reasoning problems. At a fixed model size, the probability of correctly solving an instance decays approximately exponentially with instance size, our proxy for problem hardness. The decay scale grows sublinearly with model size, indicating that larger models are more capable, but that capability gains diminish with scale. Output length grows as a power law with instance size, which serves as a proxy for difficulty. However, the parameters of this power law do not vary systematically with model size, suggesting that larger models do not become more efficient. Together, these findings reveal potential limitations of naive scaling as a strategy for developing more capable AI systems: capability improves with diminishing returns, while efficiency shows little to no improvement.

I. INTRODUCTION

The paper examines how model size relates to capability and token efficiency in reasoning, finding that capability improves with diminishing returns while efficiency shows little systematic change. It uses hierarchical Bayesian models to infer these traits from correct answers and output lengths across controlled-hardness tasks.

I. INTRODUCTION

  • Inference-time compute has a marginal cost for every query, motivating study of how its performance gains depend on model size and how resources are allocated across the model life cycle.The paper uses “scaling” broadly for resource–performance relationships, not only power laws.Source: §I · pp. 1–2
  • Five DeepSeek-R1-Distill model sizes, up to 70 B parameters, are evaluated on more than 4,000 reasoning instances per size.The experiments cover four reasoning-task classes.Source: §I · p. 2
  • Capability increases with model size, but efficiency shows little evidence of improvement: correct-solution probability declines exponentially with instance size, while output length grows as a power law whose parameters vary only weakly with model size.The authors interpret the capability increase as helping larger models solve harder instances, with diminishing gains as size grows.Source: §I · p. 2

A. Evaluation Strategy

  • Capability and efficiency are treated as latent traits inferred, respectively, from correct-solution counts and output lengths using hierarchical Bayesian models.The evaluation focuses on the number of correctly solved instances and average output length among correct responses.Source: A. Evaluation Strategy · pp. 2–3
  • The four randomly generated tasks—addition, brackets, parity, and index—use instance size as a controllable proxy for hardness and include problems with differing complexity characteristics.The tasks are designed to permit efficient verification of model answers.Source: A. Evaluation Strategy · p. 2, Figure 1 · p. 3

B. Capability

Across the four reasoning tasks, the characteristic instance-size scale for correct solutions grows sublinearly with model size. At fixed model size, solving probability declines with instance size; results are robust across statistical models and sampling temperatures.

Addition task

  • For addition at τ = 0.6, doubling parameter count increases the characteristic instance-size scale by about 1.61× at the posterior mean.The inferred scale rises from about 26.4 for the 1.5 B model to 363.5 for the 70 B model.Source: B. Capability · pp. 4–5
  • At fixed model size, the probability of correctly solving an addition instance decays exponentially with instance size, and the FAM tracks observed solve fractions well.For addition, the FAM has higher expected log predictive density than the VAM (approximately −199 versus −212).Source: B. Capability · pp. 4–5

Model comparison

  • The VAM allows model-size-dependent asymptotes; for larger models the inferred large-instance asymptote is 50 correctly solved instances, but it is near zero for smaller models that often fail to answer.Both FAM and VAM remain compatible with sublinear growth of capability.Source: B. Capability · p. 5

Other tasks

  • For index and parity, posterior mean capability exponents also indicate sublinear growth across the statistical models and temperatures studied.For parity, the VAM has higher estimated predictive density, while its posterior mean exponent is lower than the FAM's.Source: B. Capability · p. 5

Capability scaling

  • Across tasks, temperatures, and statistical models, most posterior mass for β(ν) lies above zero and below one, supporting sublinear capability scaling.Which statistical model has higher estimated predictive density varies across experimental conditions.Source: B. Capability · pp. 5–6

C. Efficiency

Across most task and temperature combinations, the study finds little evidence that output-length efficiency improves with model size. Output length nevertheless increases approximately as a power law with instance size among correct responses.

Index task

  • For index at τ = 0.6, the efficiency prefactor is approximately flat with model size, with β(A) posterior mean ≈ −0.03 and a 95% credible interval that straddles zero.Among correct responses, average output length rises sublinearly with instance size; the posterior mean α ranges from 0.52 to 0.57.Source: C. Efficiency · p. 6

Efficiency scaling

  • Across tasks, models, and temperatures, there is no strong evidence that the output-length prefactor decreases systematically with model size, as expected if larger models were more efficient.The overall pattern is that efficiency usually does not increase; observed increases or decreases are weak.Source: C. Efficiency · pp. 6–7

Brackets task

  • For brackets, efficiency effects are weak and vary with temperature: β(A) is mostly small and positive at τ = 0.4 and 0.6, while its posterior covers zero at τ = 0.8.The PM is preferred in model comparison for this task.Source: C. Efficiency · p. 6

Index task

  • For index, both statistical models provide no strong evidence that efficiency scales with model size, although the PM fits better than the PEM at τ = 0.4 and 0.6, but not 0.8.Source: C. Efficiency · p. 6

Parity and addition tasks

  • For parity, the β(A) posterior covers zero under both models but places more mass on small negative values, compatible with weak efficiency gains as models grow.Across the addition task, both models' β(A) posteriors are centered around zero.Source: C. Efficiency · p. 6

III. DISCUSSION

The discussion concludes that larger models solve harder problems but do not produce systematically shorter correct answers. Capability gains are sublinear, and the findings' scope and the separate evaluation of capability and efficiency constrain what can be concluded.

Implications

  • Larger models are more capable, but the sublinear scaling of capability means parameter increases bring diminishing returns in the characteristic problem-size scale.The authors offer sublinear growth of relevant model resources or a slowly improving ability among several as possible explanations.Source: §III · p. 7
  • The study finds no strong evidence that larger models produce shorter correct answers, so parameter investment does not systematically reduce answer length.The authors note that this could affect how quickly pretraining costs are amortized; possible explanations include algorithmic operation requirements or token efficiency not being explicitly rewarded in post-training.Source: §III · p. 7

Measurement

  • Capability and efficiency are treated as latent properties inferred from behavioral data rather than directly observed traits.Source: §III · p. 7

Limitations

  • Generalization beyond DeepSeek-R1-Distill and the four tested reasoning tasks remains an open question; capability and efficiency were also evaluated separately.The authors identify possible tokenizer confounding from distillation and call for analyses of trait interplay and life-cycle sources of gains.Source: §III · p. 8

A. Reasoning Models and Sampling

This section describes the evaluated DeepSeek-R1-Distill models and sampling setup, then introduces the reasoning tasks and the construction and computational properties of addition instances.

A. Reasoning Models and Sampling

  • The evaluation uses five DeepSeek-R1-Distill models spanning 1.5B–70B parameters, sampled at temperatures 0.4, 0.6, and 0.8.The models are distilled from the 671B-parameter DeepSeek-R1 model.Source: A. Reasoning Models and Sampling · p. 8

B. Reasoning Tasks

1. Addition

  • Addition instances sum n integers sampled with replacement from 0–100; sequential accumulation takes Θ(n) time and O(1) space, and no faster sequential algorithm exists.Source: §1 · pp. 8–9
  • Addition is conjectured to be BAPO-hard if constant-bandwidth BAPO for binary addition could solve majority, which prior work establishes as BAPO-hard.The passage states that this implication depends on the conjecture.Source: §1 · p. 9

2. Brackets

The brackets task tests whether sequences are properly nested, while parity tests whether a binary string contains an even or odd number of ones. The section also describes instance generation and algorithmic properties for both tasks.

2. Brackets

  • Bracket instances ask whether a sequence of n opening and closing symbols from three bracket types is properly nested, equivalently a member of the three-type Dyck language.Source: §2 · p. 9
  • Bracket generation targets sequences that are properly nested in expectation in 50% of cases, and the authors report evidence that models do not exploit the deletion shortcut.A stack-based decision algorithm takes O(n) time and up to O(n) space.Source: §2 · p. 9
  • Brackets' BAPO-hardness remains conjectural; establishing it rigorously is outside this paper's scope.Source: §2 · p. 10

3. Parity

  • Parity instances are random binary strings, and the answer is computed by addition modulo 2; a running parity bit gives Θ(n) time and O(1) space.Source: §3 · p. 10
  • Unlike addition, parity is BAPO-easy according to Tomlinson et al..Source: §3 · p. 10

4. Index

This section defines index instances and their computational properties, specifies how model answers are extracted, and describes components of the Variable Asymptote Model for capability.

4. Index

  • An index instance pairs a length-n integer sequence with a uniformly sampled valid position; the answer is the value at that position, and index is BAPO-easy.For bounded-size elements, space is O(1); lookup time ranges from O(1) for arrays to O(n) for linked lists.Source: §4 · p. 10

C. Extracting Answers

  • Answers are extracted only after a completed CoT, from text following </think>, by matching a boxed answer in task-specific formats.An incomplete CoT is counted as incorrect; extraction accepts specified variants such as boxed text answers and digit-grouping commas.Source: C. Extracting Answers · pp. 10–11

1. Variable Asymptote Model

  • The Variable Asymptote Model uses a binomial likelihood for correct-instance counts and a latent relationship to model capability scaling with parameter count.The supplied passage defines σ as the sigmoid function and N(μ̃, σ̃) as a normal distribution.Source: §1 · p. 11

2. Fixed Asymptote Model

The Fixed Asymptote Model constrains the asymptotic solve probability, while the Prefactor Exponent Model represents output length among correct answers with a probabilistic scaling model.

2. Fixed Asymptote Model

  • The Fixed Asymptote Model prevents the asymptotic solve probability from varying with model size and retains the Variable Asymptote Model's parametric assumptions on capability.Source: §2 · p. 11

3. Prefactor Exponent Model

  • The Prefactor Exponent Model uses a Gaussian likelihood for the logarithm of average output length among correct answers.Source: §3 · p. 11
  • Its output-length parameters depend on prefactor and exponent scaling relationships, with priors specified for their parameters and a half-normal prior for ζτ.Source: §3 · p. 11

4. Prefactor Model

The Prefactor Model retains the PEM priors but removes latent scaling of the exponent, leaving model-size variation to the random effect; the appendices report sampling diagnostics, predictive assessment, and AI use.

4. Prefactor Model

  • The Prefactor Model (PM) uses the PEM priors but omits latent scaling of α_N,τ, so model-size variation is attributed to the random effect and the τ-specific prior is omitted.This is the stated simplification of Eq. (11).Source: §4 · p. 12

E. HMC Hyperparameters and Diagnostics

  • Six HMC chains used 1,000 warm-up and 5,000 post-warmup iterations, with no divergences and R̂ < 1.01 across the reported full-data fits.The sampler used NUTS, target acceptance probability 0.99, and maximum tree depth 20.Source: E. HMC Hyperparameters and Diagnostics · p. 12
  • Out-of-sample predictive performance was assessed by leaving out each model-size and instance-size combination at a temperature, refitting, and evaluating held-out predictive density.The reported metric is leave-one-out expected log predictive density (elpd).Source: E. HMC Hyperparameters and Diagnostics · p. 12

VII. AI USE

  • The authors report using AI assistants for proofreading, mathematical derivations, and software development, and state that they validated the code.They say they verified the mathematical derivations.Source: §VII · p. 12

Appendix A: Likelihood Variance in PM and PEM

The appendix derives a first-order approximation for likelihood variance and reports sublinear capability scaling across tasks, with relatively small deviations from the latent scaling law.

Appendix A: Likelihood Variance in PM and PEM

  • A first-order Taylor expansion approximates the mean and variance of f(average L); in PM and PEM, f is the natural logarithm.The variance approximation uses the squared first derivative of f multiplied by the variance of the arithmetic mean.Source: Appendix A · p. 15

Appendix B: Additional Results

  • Across tasks and temperatures, results support sublinear latent capability scaling irrespective of statistical model, with relatively small deviations from the latent law.The appendix reports 95% credible intervals for the additional results.Source: Appendix B · p. 16

1. Capability

  • Capability scales sublinearly across the addition, brackets, index, and parity tasks; typical solvable-instance size rises from ca. 20 at 1.5 B to several hundred at 70 B parameters.The appendix reports task- and temperature-specific capability results and describes this overall pattern as sublinear.Source: §1 · p. 16, Table A1 · p. 16
  • Capability-model comparisons favor FAM for addition and index, but VAM for brackets and parity, across temperatures.Both models support sublinear scaling for the brackets, index, and parity tasks; model-specific estimated exponents differ for brackets and parity.Source: §1 · p. 16

2. Efficiency

For the studied family and four tasks, efficiency scaling is weak or absent: output length grows sublinearly with instance size, while its power-law parameters show little systematic change with model size.

2. Efficiency

  • |E[β(A)_τ | D]| ≤ 0.1 for addition under PM; PEM estimates weakly negative scaling for both prefactor and exponent, with the 95% credible interval spanning positive and negative values.PEM is preferred in model comparison for addition.Source: §2 · p. 16
  • For brackets, PM estimates slightly positive prefactor scaling at τ = 0.4 and 0.6, and near-zero scaling at τ = 0.8; similar PEM values are reported, with PM preferred.The PEM’s exponent-scaling estimates are close to zero.Source: §2 · p. 17
  • For index, both models estimate small prefactor scaling (|E[β(A)_τ | D]| ≤ 0.05); PM is favored at τ = 0.4 and 0.6, while PEM is favored at τ = 0.8.The reported 95% credible intervals for the scaling parameters cover positive and negative values.Source: §2 · p. 17
  • For parity, the expected prefactor-scaling exponent lies between −0.17 and −0.12, but its 95% credible interval includes positive and negative values; PEM gives no strong evidence of exponent scaling..Source: §2 · p. 17
  • 0.41–0.89: the posterior mean exponent α_N,τ indicates sublinear output-length scaling with instance size; prefactors are largest for addition and smallest for index.The prefactor pattern holds irrespective of statistical model; the table caption says prefactors differ by task but less by temperature.Source: §2 · p. 17, Table A4 · p. 19
  • Efficiency scaling is weak or absent across the four tasks: prefactor scaling is close to zero, and the PEM’s exponent-scaling estimates are low with intervals spanning zero.The authors report this pattern for both efficiency models and note that random-effect standard deviations tend to be small.Source: §2 · pp. 17–18

Appendix C: Model Comparison

  • Efficiency-model elpd differences are generally much smaller than the task-dependent preferences: PEM is favored for addition, while PM is slightly preferred for brackets and parity.For index, preference shifts from PM at lower temperatures to PEM at τ = 0.8.Source: Appendix C · p. 18, Table A5 · p. 19

Appendix D: Brackets

  • The frequent appearance of “stack” or “Stack” in brackets answers, especially for larger models, is consistent with stack-based reasoning rather than a length-based shortcut.The authors say inspected answers corroborate this interpretation.Source: Appendix D · p. 18, Figure A1 · p. 20

Related papers

Loading 2609.27166v2…