Source-linked AI summary
A Simple Model of Inference Scaling Laws
Noam Levi
TL;DR
Inference scaling through repeated attempts has received less attention than scaling with model size, data, and compute. The paper develops a statistical toy model for coverage and inference loss, then reports predictions consistent with observed LLM behavior and a simple VAE construction.
Problem
Inference scaling through repeated attempts has received less attention, leaving open how to interpret or predict model performance as attempts increase.
Source: §1 · p. 1Method
Results
The predicted coverage behavior matches observed LLM repeated-trial behavior, and the predictions are verified using prior LLM results and a simple generative VAE construction.
Source: §1 · pp. 1–2, §5 · p. 9Takeaways & Limitations
The framework offers a starting point for studying trade-offs between training and inference attempts, including performance and cost.
Source: §1 · p. 2, §5 · p. 9The simple constructions are not unique, and extending the framework to more complex models and its interaction with existing scaling laws remains future work.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Neural scaling laws have garnered significant interest due to their ability to predict model performance as a function of increasing parameters, data, and compute. In this work, we propose a simple statistical ansatz based on memorization to study scaling laws in the context of inference, specifically how performance improves with multiple inference attempts. We explore the coverage, or pass@k metric, which measures the chance of success over repeated attempts and provide a motivation for the observed functional form of the inference scaling behavior of the coverage in large language models (LLMs) on reasoning tasks. We then define an "inference loss", which exhibits a power law decay as the number of trials increases, and connect this result with prompting costs. We further test our construction by conducting experiments on a simple generative model, and find that our predictions are in agreement with the empirical coverage curves in a controlled setting. Our simple framework sets the ground for incorporating inference scaling with other known scaling laws.
1 Introduction
Neural scaling laws describe predictable performance changes with model size, data, and compute, while inference scaling through repeated attempts has received less attention. This work proposes a toy model for predicting coverage under repeated inference and connects it to inference cost and loss.
1 Introduction
- Neural scaling laws describe predictable performance changes as model size, data size, and computational resources increase.The paper situates inference scaling alongside these established dimensions.Source: §1 · p. 1
- Repeated prompts have been reported to improve weaker models on difficult coding and formal-proof tasks without further training.Those tasks allow verification of correct answers.Source: §1 · p. 1
- The paper proposes a toy model to predict how coverage, or pass@k, changes with the number of inference attempts.The motivation is to interpret or predict inference scaling under repeated attempts.Source: §1 · p. 1
- The paper presents analytical coverage predictions that match observed repeated-trial behavior in prior LLM studies.It also defines inference loss and connects coverage with total inference cost.Source: §1 · pp. 1–2
- The predictions are tested with a VAE trained to reconstruct its training data by sampling from a latent space with an associated temperature.The paper frames this construction as a controlled generative setting, isolated from other neural scaling laws.Source: §1 · p. 2
2 Related Work
Prior scaling-law research has largely studied training, with empirical work on neural networks and analyses of several solvable model classes. The paper positions inference scaling as comparatively under-explored.
2 Related Work
- Empirical scaling research reports predictable decreases in error rates as data, parameters, and compute increase, including language and multimodal models.Kaplan et al. [2020b] studied language models, while Henighan et al. [2020b] extended scaling-law work to multimodal models.Source: §2 · p. 2
- Other studies examine scaling in solvable models, from generalized linear regression on random features to kernel ridge regression, connecting results to random matrix theory and data properties.The passage cites Cabannes et al. [2024], Maloney et al. [2022], Bordelon et al. [2020], Spigler et al. [2020], Caponnetto and De Vito [2007], Steinwart et al. [2009], Fischer and Steinwart [2020], Cui et al. [2021], Levi and Oz [2023, 2024], and Nam et al. [2024].Source: §2 · p. 2
- The paper identifies inference scaling—how performance changes with repeated attempts—as less explored than training-focused scaling laws.Source: §2 · p. 2
3 Memorizing Ansatz
The section uses memorization-based models to describe how repeated inference attempts affect coverage and loss. It fits LLM pass@k curves, interprets task difficulty through failure probabilities, and extends the account to correlated trials and inference cost.
3 Memorizing Ansatz
- The setup adapts the Hutter memorization model, which assumes perfectly memorized training features and Zipf-distributed feature frequencies, to study repeated inference attempts.The authors use memorization as a surrogate for training while shifting attention to sample-level inference success.Source: §3 · p. 2
3.1 Perfect Sample Memorization, Imperfect Predictions
- The model assumes perfect sample memorization but imperfect inference, and estimates pass@k as the chance of at least one success across k independent attempts with perfect verification.It averages over samples with potentially different failure probabilities.Source: §3.1 · pp. 2–3
3.2 Power Law Distributions Predict Pass@k for LLMs
- The Beta-distribution pass@k form in Eq. (6) approximates observed LLM curves on mathematical and coding tasks.Figure 1 compares empirical curves with the analytical predictions.Source: §3.2 · p. 4, Figure 1 · p. 4
- At large k, inference loss decays as a power law whose exponent β controls how many attempts harder problems require.The asymptotic form is proportional to k^-β.Source: §3.2 · p. 4
3.2.1 Interpretability
- Fits suggest β ∼ 0.35 and 2 < α < 20, indicating mostly difficult questions with an easy-sample tail covered in fewer than 100 trials.The mean and concentration of the Beta distribution describe average failure probability and homogeneity in difficulty.Source: §3.2.1 · p. 5
- Interpreting the pass@k average as a Laplace transform suggests that its inverse can recover perceived-difficulty distributions and help select hard but obtainable training samples.The proposed fine-tuning strategy biases selection toward difficult but obtainable tasks.Source: §3.2.1 · pp. 5–6
3.3 Correlated Trials and Effective k Approach
- A correlated-trial model replaces k with an effective independent-trial count based on power-law decay in trial correlations, and approximates LLM behavior for k ≫ 1.The authors describe this as an effective account of sample correlations.Source: §3.3 · p. 6
3.4 Connection to Compute Scaling
- Inference loss also decays as a power law with total inference cost, with the decay depending on β.The cost model treats prompt and decode tokens and FLOPS per token as components of total cost.Source: §3.4 · pp. 6–7
4 Experiments on a Simple Generative Model
The VAE reconstruction experiment tests whether the effective-trial model predicts pass@k when reconstructions are sampled repeatedly. Its predictions approximate the measured coverage curves, while trial-error correlations show why attempts cannot be treated as independent.
4 Experiments on a Simple Generative Model
- The experiment trains a temperature-controlled VAE to reconstruct Fashion-MNIST examples treated as memorized samples.The setup aims to connect the theoretical memorization model with behavior in a more complex generative model.Source: §4 · p. 7
- Effective-k predictions approximate the VAE pass@k curves across reconstruction-error thresholds.The figure also shows reconstruction errors across multiple attempts; the predictions use empirical failure probabilities at maximal k and κ from the correlation matrix.Source: Figure 5 · p. 8, §4 · p. 8
- Pass@k is the fraction of input samples with at least one reconstruction error below threshold ϵ across k trials.Per-sample error is defined from the difference between the original image and its reconstruction.Source: §4 · pp. 7–8
- Error-correlation eigenvalues decay as a power law, indicating that the effective number of independent trials diminishes as k increases.The correlation matrix shows clear correlations between trials at k = 1000.Source: §4 · pp. 8–9, Figure 6 · p. 9
5 Conclusion
The conclusion presents a simple statistical account of inference scaling in repeated-attempt coverage. It derives pass@k and inference-loss predictions, verifies them against LLM results and a VAE, and frames the construction as one of many possible models.
5 Conclusion
- Two constructions—a distribution of easy and difficult problems and a Zipf-like trial-correlation structure—yield analytical predictions for pass@k and inference loss.The paper reports empirical verification using previous LLM results and a simple generative VAE.Source: §5 · p. 9
- The same functional behavior can arise from many models, including alternatives based on failure distributions or program lengths.The authors present the construction’s simplicity as a strength and interpret the behavior in terms of task complexity relative to the model.Source: §5 · p. 9
- The authors propose extending the phenomenological framework to more complex models and studying its interaction with scaling laws for model size and training data.Suggested settings include generalized linear regression, kernel regression, and neural networks.Source: §5 · p. 9
A Experimental Setup
The appendix specifies the VAE architecture, training procedure, and qualitative inspection used for the Fashion-MNIST reconstruction experiment.
A Experimental Setup
- The VAE uses 784-dimensional inputs, a 400-unit hidden layer, a 20-dimensional latent space, and a two-layer decoder with sigmoid output.Temperature T = 1.1 is applied during reparameterization to control latent-variable variance.Source: A Experimental Setup · p. 12
- Training uses the first 400 Fashion-MNIST samples for 1000 epochs with binary cross-entropy plus KL divergence and Adam at learning rate 1 × 10^-3.Source: A Experimental Setup · p. 12
- Figure 7 presents Fashion-MNIST inputs beside their reconstructions for qualitative inspection of successful and failed cases and error types.Source: A Experimental Setup · p. 12, Figure 7 · p. 12