PaperlayerView exact revision on arXiv

Source-linked AI summary

When does a scaling result justify a different allocation? A critical review of resource-allocation evidence for AI systems

Seyed Morteza Emadi

arXiv:2609.14500v1cs.AIcs.LG

TL;DR

AI scaling scores do not by themselves show where additional resources should be spent, because deployed systems make decisions about computation, tools, retrieval, and stopping. This review compares allocation evidence, distinguishing tested procedures from feasible frontiers and identifying recurring mismatches in success measurement, information access, and costs. It proposes a resource envelope and concludes that allocation decisions require deployment-aligned measures of all three.

  • Problem

    Scaling evidence often evaluates a system under success measures, information access, or costs that may differ from deployment, complicating resource-allocation comparisons.

    Source: §1 · pp. 1–2

  • Method

    The review compares scaling evidence using budgets, mechanisms, and information, distinguishes tested procedures from attainable frontiers, and proposes a resource envelope for reporting results.

    Source: §1 · p. 2, §3.1 · p. 6, §3.2 · p. 7

  • Results

    The review identifies three recurring mismatches: success counted before an answer is chosen, information unavailable in deployment, and costs omitted from comparisons.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A result supports a different allocation when success is measured on the returned answer, information matches deployment, and costs are counted at the team’s boundary.

    Source: §9 · p. 26

    The purposive search is not exhaustive, the evidence centers on language-model systems, heterogeneous studies preclude a pooled scaling coefficient, and analytical examples do not estimate effects or optimal policies.

    Source: §9 · p. 26

Abstract

from arXiv · show

AI scaling studies increasingly evaluate systems that combine a pretrained model with retrieval, search, verification, tools, and interaction. Yet a higher score under a larger budget does not by itself show where additional resources are best spent. This critical integrative review asks when a reported scaling result supports a resource-allocation decision. It compares evidence across pretraining, test-time computation, retrieval, and agent evaluation, distinguishing the performance of a tested procedure from the best performance achievable under a resource limit. The synthesis shows that three mismatches recur across this evidence: success counted before an answer is chosen, information a deployed system will not have, and costs left out of the comparison. A capability surface expresses performance as a function of budgets, mechanisms, and available information. Worked analytical examples show how the evaluation metric, deployment volume, selection rule, and stopping policy can alter an allocation conclusion. A resource envelope provides a structured record of the task, development and run-time resources, information access, and procedure behind a reported score. Its application to a published comparison illustrates which conclusions the evidence supports and which deployment questions remain unresolved. The resulting framework specifies the comparisons needed to choose among feasible systems and motivates experiments on the transfer of allocation rules across tasks and operating conditions. It does not propose a universal scaling law or infer general intelligence from benchmark gains.

1 Introduction

The review asks when a scaling result supports reallocating resources, given that deployed systems choose how to spend budgets on building and running. It argues that scores can mislead allocation decisions when success, information access, or costs differ from deployment, and proposes a resource envelope to make those conditions visible.

1 Introduction

  • An allocation divides a build-and-run budget to meet a task’s quality requirement at lowest total cost using only deployment-available information.A scaling result measures how quality responds to one resource while other settings are held fixed; an allocation claim asserts that one budget division is better than another.Source: §1 · p. 2
  • Repeated sampling may raise coverage, but users receive one answer, so returned-answer accuracy depends on selecting a correct candidate.Coverage counts a question as solved if any generated answer is correct; returned-answer accuracy counts whether the one answer returned is correct.Source: §1 · p. 2
  • A result using an answer key to allocate computation or select an answer may not transfer to deployment, where no answer key is available.Source: §1 · p. 2
  • A study’s cost may omit the verifier’s training and measure generation in FLOPs, while a team pays dollars and latency.Source: §1 · p. 2
  • The review distinguishes evidence that a procedure improves with more resources from evidence that it beats alternatives at equal cost or establishes claims about untested alternatives.It proposes a resource envelope that records the score, costs, and information available to the system, so readers can identify mismatches before acting.Source: §1 · p. 2

2 Review approach and relation to existing syntheses

The review uses a purposive, non-systematic search and reads studies for what they vary, measure, hold fixed, or omit. It builds on existing syntheses by asking whether each result supports choosing between ways to spend the same budget.

2.1 Scope, search, and evidence handling

  • The review is not a systematic review or meta-analysis, and its purposive search yields a chosen sample rather than a census or pooled effects.Its search closed on 9 September 2026 and used foundational papers, citation tracking, and targeted searches across specified topics and sources.Source: §2.1 · pp. 3–4
  • Studies were included when they varied a resource or mechanism, analyzed a computational or information constraint, or stated an evaluation principle needed to interpret variation.For evidence tables, the review records what was varied, measured, held fixed, or left out, checking relevant cited-edition passages when methods or accounting mattered.Source: §2 · p. 3, §2.1 · p. 4

2.2 What existing work already contributes

  • Existing work covers metareasoning, training–inference scaling, compound-system choices, and resource-aware agent evaluation; the review builds on these contributions.Source: §2.2 · p. 4
  • Prior surveys organize reasoning, test-time scaling, sampling, search, trajectory optimization, and verification; this review adds an allocation question applied to each result.It asks when evidence suffices to choose between ways of spending the same budget, without implying earlier surveys ignore cost or systems.Source: §2.2 · p. 5, Table 2 · p. 5

3 A framework for interpreting system scaling

The framework reads system quality in relation to budgets, mechanisms, and information, while distinguishing the performance of a tested procedure from the best feasible performance under a budget. It also makes accounting boundaries and score choice explicit because they affect allocation comparisons.

3.1 Budgets, mechanisms, and information

  • A system’s quality depends on what it spends, how it spends it, and what information it may use; these are budgets, mechanisms, and information.Budgets can include training, inference, storage, tool calls, and environmental steps; mechanisms include retrieval, verification, coordination, and stopping policies.Source: §3.1 · p. 6
  • The feasible set reflects dependencies and deployment constraints: memory can rule out a model, and deadlines can prevent serial search.For a specified dense training recipe, training compute is approximately proportional to parameter count times processed tokens, so these are not independent axes.Source: §3.1 · p. 6
  • A pretraining scaling study holds acquisition policy, inference policy, and system configuration fixed, making its result interpretable while leaving returns to changing them unmeasured.Source: §3.1 · p. 6

3.2 Performance curves, frontiers, and what a budget buys

  • The attainable frontier is the least upper bound on feasible quality under a budget, whereas a tested procedure can deteriorate as resources increase without showing that every use of those resources is harmful.The best tested configuration is not globally optimal without an appropriate proof or search guarantee.Source: §3.2 · p. 7
  • Lifecycle comparisons depend on the accounting boundary and deployment volume; FLOPs, tokens, money, energy, and latency are not interchangeable without assumptions.Development can include training, verifier construction, indexing, and configuration search; shared pretraining may be treated as sunk for one deployment decision or charged for an end-to-end comparison.Source: §3.2 · p. 7
  • Efficiency interventions can shift expenditure between development and run-time or change feasible hardware, so comparisons should specify compression and serving conditions.Distillation does not guarantee preservation of every capability; quantization comparisons should specify precision, calibration data, kernels, and compressed-model production costs.Source: §3.2 · p. 7

3.3 Interactions depend on the outcome being measured

  • The sign of a resource interaction can change under an increasing transformation of the score, even when that transformation preserves allocation rankings.Cost to achieve a fixed performance level can provide a more decision-relevant comparison; objectives such as perplexity and classification accuracy can rank systems differently.Source: §3.3 · p. 8
  • Changing a retriever or verifier changes the attainable surface, while new observations change available information; neither is simply movement along a fixed curve.Source: §3.3 · p. 8

4 Critical synthesis of the scaling evidence

The review distinguishes evidence that a procedure improves from evidence that identifies a mechanism, compares tested systems, or establishes a feasible frontier. Across pretraining, inference, retrieval, verification, and agent studies, allocation conclusions remain conditional on the metric, information, costs, and procedures evaluated.

4 Critical synthesis of the scaling evidence

  • The framework asks whether a procedure improves, what mechanism accounts for the change, which tested system is preferable, and what can be established about the best feasible performance.Each question requires evidence suited to its claim; a study may answer one without answering the others.Source: §4 · p. 8

4.1 Pretraining and the split between development and run-time

  • Sardana et al. (2024) favor smaller, longer-trained models under substantial inference demand, but the review finds no universal balance or common exponent for training and inference.Pretraining evidence also depends on the training recipe and extrapolation range; unique information, processed tokens, and repeated exposure are distinct resources, and downstream transfer needs separate measurement.Source: §4.1 · p. 9

4.2 Test-time computation is a policy, not just a token count

  • Test-time allocation depends on the inference policy: tested preferences vary with prompt difficulty and configuration, while more calls can produce non-monotone aggregate accuracy.Serial depth, branching, and parallelism differ in cost; selective reasoning and stopping approaches depend on estimates of difficulty, sample value, or reward calibration.Source: §4.2 · pp. 9–10

4.3 Retrieval changes both accessibility and resource accounting

  • Retrieval gains can reflect newly exposed information, better access to encountered information, or computation on evidence, so experiments need controls that separate these explanations.Datastore-scale gains and reported token-allocation patterns do not alone establish lifecycle savings; indexing, retrieval, context processing, storage, and workload assumptions matter.Source: §4.3 · pp. 11–12

4.4 Generation, verification, and the returned answer

  • Repeated sampling measures coverage, but deployment also depends on whether a selector returns a correct answer; optimization can shift verifier behavior and increase reward hacking.Verification results also depend on supervision and feedback source, and verifier development and training belong in allocation comparisons.Source: §4.4 · pp. 12–13

4.5 Interaction, coordination, and evaluation infrastructure

  • Agent results depend on interaction and coordination mechanisms, while detailed cost tracking may cover fewer configurations than quality evaluation.Cumulative success under oracle-feedback or no-feedback protocols is not automatically final-answer accuracy, and reported token accounting may exclude the judging model.Source: §4.5 · pp. 13–14

4.6 What the evidence jointly supports

  • Across the reviewed literatures, a resource helps only through a mechanism that uses it, and allocation claims must distinguish conditional performance, mechanism, tested-system preference, and the feasible frontier.The next experiment depends on the claim: controlled intervention, more cost measurement or held-out selection, or a guarantee extending beyond tested configurations.Source: §4.6 · pp. 14–15

5 Capability bottlenecks: computation, information, and state

Task analysis links what limits performance—validation, identifying evidence, or retaining intermediate state—to which resource intervention to test next. These task categories can overlap rather than partitioning tasks.

5.1 Discovery requires a claim about validation

  • Discovery separates finding a candidate from validating it: more sampling targets weak generators, while better supervision or checking targets weak selectors and design changes may reduce checker cost.Measure the complete acceptance procedure, since candidate generation paired with an unavailable oracle estimates an opportunity for further work, not deployable performance.Source: §5.1 · p. 15

5.2 Causal identification is an information constraint

  • At least (ρx)^2/4 worst-case mean squared error remains for the interventional mean under observational samples alone; lower error requires information or assumptions that distinguish the two worlds.More retrieval from the same observational distribution may reduce estimation error without resolving ambiguity; a valid experimental report can change identification.Source: §5.2 · p. 16

5.3 Sequential composition depends on accessible computation and state

  • Sequential tasks depend on intermediate results remaining accessible; relevant resources include depth, generated steps, recurrent state, context, and external memory.Test task depth alongside resources: token count alone does not establish useful sequential computation, and memory or adaptive inference is not universally required.Source: §5.3 · pp. 16–17

5 Capability bottlenecks: computation, information, and state

  • Table 6 frames task demands as candidate resource interventions under conditions, and the task categories may overlap rather than partitioning tasks.The proposed interventions are not universal prerequisites.Source: Table 6 · p. 17, §5.3 · p. 17

6 Worked analytical examples

Three analytical examples show that allocation conclusions depend on the score, selection and stopping rules, deployment volume, and costs. Their probabilities and costs are stipulated for illustration, not fitted to reviewed studies or deployed-system performance.

6.1 Retrieval: metric choice and deployment volume

  • Retrieval reduces perplexity by approximately 0.635 for the small model and 0.223 for the large model, but improves accuracy only for the large model.The example uses hypothetical probabilities; the small model gets the larger perplexity improvement, while only the large model crosses the classification decision threshold.Source: Table 7 · p. 17, §6.1 · p. 18
  • At 68,000 queries, the illustrative lifecycle costs cross: retrieval is cheaper below that volume, while the parameter-heavy system is cheaper above it.The assumed systems meet equal quality, information-access, and latency requirements; the retrieval system's development cost includes its index.Source: Figure 2 · p. 18, §6.1 · p. 18

6.2 Generation and selection under the same allowance

  • With a 12-unit allowance, the stronger selector returns more accurate answers with six candidates than the cheap selector does with ten, despite lower coverage.The stronger selector reaches returned-answer accuracy 0.701 with coverage 0.738; the cheap selector reaches 0.580 with coverage 0.893.Source: §6.2 · p. 19
  • Under the constant-selection-accuracy assumption, the cheap selector cannot reach 0.70 returned-answer accuracy at any sample count, while the stronger selector reaches it with six candidates.The assumed ceiling is specific to that selector; in practice, conditional selection accuracy may change with set size and should be estimated on representative tasks.Source: §6.2 · p. 19

6.3 Stopping, abstention, and wrong answers

  • Under the stipulated checker errors, increasing the trial cap reduces abstention but raises both correct and wrong returns, leaving conditional reliability unchanged.For p = 0.2, s = 0.9, and f = 0.05, conditional reliability is about 0.818; a benchmark treating abstentions and wrong answers alike may favor more correct returns.Source: §6.3 · p. 19, Figure 3 · p. 20
  • When a wrong return incurs a loss, more trials can reduce expected utility even if they increase correct returns; adaptive policies require a richer model than homogeneous trials.For the stated probabilities, v = 1, ℓ = 5, and c = 0, the expected net gain from another attempt is −0.02; adaptive generation, correlated errors, and changing checker calibration violate the assumptions.Source: §6.3 · p. 20

7 A resource envelope for comparable evaluation

The resource envelope records the budgets, mechanisms, information, and procedure behind a score so readers can assess what a comparison establishes. The section distinguishes controlled component comparisons from deployment choices and outlines experiments for estimating resource interactions.

7.1 The score and the process that produced it

  • The resource envelope is a structured record of how a result was produced, covering development and run-time resources, information access, and selection.Reports should mark unavailable quantities as unknown or deliberately excluded, and state how each omission affects the comparison.Source: §7 · p. 21, §7.1 · p. 21, Table 8 · p. 21
  • In the Snell et al. (2025) comparison, the outcome is single-answer accuracy, but the headline difficulty sorting uses the test answer key; the verifier-based version uses deployment-available information.The study reports up to four times less inference compute than a best-of-N baseline at equal accuracy, but does not report the quantities needed to determine whether omitted costs reverse the saving.Source: Table 9 · p. 23, §7.1 · p. 24

7.2 Two legitimate comparison questions

  • Controlled comparisons isolate a resource or component by holding the remaining procedure fixed, whereas deployment comparisons choose the best tested system under comparable tuning and held-out evaluation.The latter supports system choice but cannot attribute the difference to model size alone; reporting both views can separate experimental control from practical optimization.Source: §7.2 · p. 24

7.3 Estimating resource interactions

  • Resource-interaction experiments should use repeated runs and task-level uncertainty, with more budget levels for curvature and separate treatment of mechanism or information changes.When adaptation matters, evaluate allocation policies across a query stream and report the task mixtures used for learning and evaluation.Source: §7.3 · pp. 24–25

8 Research agenda: from local curves to transferable decisions

The agenda calls for experiments that test whether allocation rules transfer across scales and operating conditions. It specifies comparisons for development and run-time trade-offs, information access, verification, interaction, and workload uncertainty, while leaving outcomes open.

8 Research agenda: from local curves to transferable decisions

  • Fit development-versus-run-time trade-offs on returned-answer accuracy and actual cost, then test allocation rankings on withheld scales and deployment volumes.The proposed experiments jointly vary model size, training duration, and inference policy, reporting coverage where relevant.Source: §8 · p. 25
  • Cross model scale with corpus size, overlap, retriever quality, and processing budget to distinguish new information from better access or more computation.Include reused training data, distribution-matched new data, and genuinely new task-relevant information.Source: §8 · p. 25
  • Evaluate verifiers on candidates from large or adaptive searches, tracking selection accuracy, abstention, wrong returns, proposal correlation, and development cost.The comparison should predict when selection is more effective than additional proposals under a common total allowance.Source: §8 · p. 25
  • Cross agent horizon with training regime and per-step deliberation, control tools and environment versions, and test whether gains persist on longer or changed tasks.Record observations and resets, and compare serial and parallel coordination under work and latency constraints.Source: §8 · p. 25
  • Test allocation policies under shifts in demand, query difficulty, failure costs, and prices, measuring regret or excess cost against a defined feasible comparator.Include tail constraints and the expense of adapting or tuning a policy.Source: §8 · pp. 25–26
  • The proposed comparisons are falsifiable rather than predictive: retrieval may lose cost-effectiveness after serving costs, and longer interaction may add unproductive actions.Other possibilities include verification failing under candidate shift and small-model inference advantages disappearing when useful proposals are rare.Source: §8 · p. 26

9 Limitations and conclusion

A scaling result supports a different allocation when success is measured on the returned answer, information access matches deployment, and costs are counted at the team’s boundary. The review’s evidence is limited in scope and does not yield a pooled coefficient or an optimal deployment policy.

9 Limitations and conclusion

  • The purposive search is not exhaustive and time-bounded, the empirical center is language-model systems, and heterogeneous studies do not support a pooled scaling coefficient.The analytical examples use simplifying assumptions and establish possibilities and reporting needs, not estimated effects or optimal deployment policies.Source: §9 · p. 26
  • The capability surface is high-dimensional and expensive to estimate, may rely on partially observable information, and does not itself solve the allocation optimization problem.Its utility and reporting burden still require evaluation in practice.Source: §9 · p. 26
  • The relevant unit is the complete procedure producing a scored outcome; the decision is which feasible allocation reaches the task requirement most effectively.This follows once quality, information access, and costs are aligned.Source: §9 · p. 26
  • A result justifies reallocating resources when it scores the returned answer, matches deployment information access, and counts costs at the team’s boundary.If any condition is missing, the resource envelope identifies it as an experiment to run next.Source: §9 · p. 26

A.1 Increasing transformations of the score

The argument distinguishes numerical interaction signs from trade-offs along a regular level set. A positive derivative under an increasing transformation does not guarantee that the mixed derivative keeps its sign.

A.1 Increasing transformations of the score

  • An increasing transformation can have a positive first derivative while the directly computed mixed derivative is negative.For the example, g = b1b2 and h(z) = −1/z; the passage uses this to distinguish numerical interaction signs from ordinal trade-offs.Source: §A.1 · p. 26
  • Along a regular level set, multiplication by the positive derivative h′(g) leaves the trade-off slope unchanged.The argument concerns ordinally invariant slopes, not invariant numerical interaction signs.Source: §A.1 · p. 26
  • The calculation is presented as an elementary distinction between numerical interaction signs and ordinally invariant trade-offs, not as a new mathematical result.Source: §A.1 · p. 26

A.2 Coverage and selection

The worked example compares feasible candidate counts under two cost assumptions and multiplies coverage by assumed conditional selection accuracy to obtain returned-result figures.

A.2 Coverage and selection

  • 0.5802068 and 0.7009632 are the resulting products of coverage and assumed conditional selection accuracy for feasible counts of 10 and 6, respectively.The example assumes independent candidates with common success probability p; a correct return requires at least one correct candidate in the set.Source: §A.2 · p. 27

A.3 First-accept stopping

The analytical examples show how first-accept stopping determines return and attempt probabilities, and how observationally indistinguishable causal models constrain intervention-mean estimation.

A.3 First-accept stopping

  • A correct acceptance on attempt t requires t−1 prior nonacceptances and a correct acceptance; summing over n attempts gives Eq. (10).Substituting (1−p)f gives the wrong-return probability.Source: §A.3 · p. 27
  • No acceptance has probability (1−a)^n; when a = 0, all n attempts occur, and expected attempts sum the probabilities of reaching each attempt.Source: §A.3 · p. 27

A.4 Observational equivalence and estimation error

  • The two causal models share a joint Gaussian observational law, so estimates based only on its samples and common auxiliary information have the same distribution in both.Source: §A.4 · p. 27
  • The larger of the two risks is at least their average; supplying distinguishing information changes the premise.Source: §A.4 · p. 27
Loading 2609.14500v1…