PaperlayerView exact revision on arXiv

Source-linked AI summary

Cost-of-Pass: An Economic Framework for Evaluating Language Models

Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yuksekgonul, James Zou

arXiv:2504.13359v2cs.AIcs.CL

TL;DR

Accuracy and capability alone do not show whether language models deliver value relative to inference or human-labor costs. This paper adapts production theory into cost-of-pass and frontier cost-of-pass measures, finding task-specific model-family roles and rapid frontier progress, while common test-time techniques offer limited economic benefit.

  • Problem

    Standard capability metrics omit inference cost, leaving unclear whether language models' economic benefits outweigh their costs or human labor.

    Source: §1 · pp. 1–2

  • Method

    The framework adapts production theory, measuring expected monetary cost per correct solution and the minimum such cost across language models and a human-expert baseline.

    Source: §2.2 · p. 4, §2.3 · p. 4, §2.4 · p. 4

  • Results

    Lightweight, large, and reasoning models contribute most on basic quantitative, knowledge-based, and complex quantitative tasks, respectively; complex-task frontier costs roughly halved every few months.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Recent cost-efficiency gains were mostly driven by model-level innovations; common test-time techniques offer limited or no economic benefit, while TALE-EP offers some.

    Source: §1 · p. 3

    The human-expert baseline assumes near-perfect correctness for experts qualified for the benchmark tasks.

    Source: §2.4 · p. 4

Abstract

from arXiv · show

Widespread adoption of AI systems hinges on their ability to generate economic value that outweighs their inference costs. Evaluating this tradeoff requires metrics accounting for both performance and costs. Building on production theory, we develop an economically grounded framework to evaluate language models' productivity by combining accuracy and inference cost. We formalize cost-of-pass: the expected monetary cost of generating a correct solution. We then define the frontier cost-of-pass: the minimum cost-of-pass achievable across available models or the human-expert(s), using the approx. cost of hiring an expert. Our analysis reveals distinct economic insights. First, lightweight models are most cost-effective for basic quantitative tasks, large models for knowledge-intensive ones, and reasoning models for complex quantitative problems, despite higher per-token costs. Second, tracking the frontier cost-of-pass over the past year reveals significant progress, particularly for complex quant. tasks where the cost roughly halved every few months. Third, to trace key innovations driving this progress, we examine counterfactual frontiers -- estimates of cost-efficiency without specific model classes. We find that innovations in lightweight, large, and reasoning models have been essential for pushing the frontier in basic quant., knowledge-intensive, and complex quant. tasks, respectively. Finally, we assess the cost-reductions from common inference-time techniques (majority voting and self-refinement), and a budget-aware technique (TALE-EP). We find that performance-oriented methods with marginal performance gains rarely justify the costs, while TALE-EP shows some promise. Overall, our findings underscore that complementary model-level innovations are the primary drivers of cost-efficiency and our framework provides a principled tool for measuring this progress and guiding deployment.

1 INTRODUCTION

Evaluating language models economically requires accounting for both capability and inference cost, not accuracy alone. The paper applies production theory to define cost-of-pass and its frontier, then uses them to track progress and compare model families and inference-time techniques.

Motivation

  • Standard capability metrics omit cost, leaving evaluations unable to show whether a model's delivered value justifies its inference expense.The paper motivates evaluating models within a broader ecosystem because marginal improvements may not justify higher costs.Source: §1 · pp. 1–2

Concepts

  • The framework adapts production theory to language models as stochastic producers, whose strategies can trade higher success probability for greater computational cost.The economic objective is minimizing cost per successful output under uncertainty.Source: §1 · p. 2
  • Cost-of-pass measures expected monetary cost per successful output, while frontier cost-of-pass is the minimum achievable across available models and a human-expert option.The human-expert cost baseline is included in the frontier definition.Source: §1 · pp. 2–3

Tracking progress with frontier cost-of-pass

  • Frontier cost-of-pass decreased exponentially across three task categories, with complex quantitative costs roughly halving every few months.The analysis covers May 2024 to February 2025.Source: §1 · p. 3

Counterfactual frontier in the absence of model families

  • Lightweight models are most impactful on basic quantitative tasks, large models on knowledge-based benchmarks, and reasoning models on complex quantitative challenges.The counterfactual analysis identifies these model-family contributions using basic quantitative tasks, GPQA-Diamond, AIME, and MATH as examples.Source: §1 · p. 3

Impact of post-hoc inference time techniques

  • Common inference-time techniques offer limited or no economic benefit, while budget-aware TALE-EP delivers some; recent frontier reductions are mostly driven by model-level innovations.The evaluated techniques include self-refinement and majority voting.Source: §1 · p. 3

2 SETUP

Using production theory, the paper develops measures of the cost of obtaining correct LM outputs and the minimum cost across available options, including a human-expert baseline. It then describes how to estimate these costs and track economic progress across problems and model releases.

2.1 ECONOMIC THEORY OF PRODUCTION EFFICIENCY

  • The framework adapts production theory’s minimum-cost perspective to stochastic LM producers, with correctness as the target output.Production theory frames efficiency as the minimum resource cost needed to achieve a target output.Source: §2.1 · p. 3

2.2 COST-OF-PASS: AN EFFICIENCY METRIC FOR LMS

  • Cost-of-pass combines an LM’s expected per-attempt cost and probability of success to measure the expected monetary cost of obtaining one correct solution.If the model cannot solve the problem, its cost-of-pass is infinite.Source: §2.2 · p. 4

2.3 THE LM FRONTIER COST-OF-PASS

  • The LM frontier cost-of-pass is the minimum cost-of-pass achievable across available LM strategies for a problem.It captures the best available LM strategy rather than a single model’s efficiency.Source: §2.3 · p. 4

2.4 GROUNDING EVALUATION: ESTIMATED HUMAN-EXPERT BASELINE

  • Adding a human-expert baseline makes the frontier compare LM strategies with the estimated cost of qualified human labor.The expert-cost estimate considers required expertise, time per problem, and compensation rates.Source: §2.4 · p. 4

2.5 MEASURING PROGRESS AND VALUE GAIN

  • The framework defines a new model’s gain as the reduction in frontier cost-of-pass relative to prior options, including humans.As strategies are added, the frontier cost-of-pass is non-increasing over time.Source: §2.5 · p. 5

2.6 ESTIMATING THE ECONOMIC EFFICIENCY

  • The estimation workflow measures repeated-attempt success and per-attempt costs, computes model and expert frontiers, then aggregates across benchmark problems.Per-attempt costs include prompt and generated tokens priced at current rates and any extra charges.Source: §2.6 · p. 5

2.6.1 ESTIMATING HUMAN-EXPERT COST

  • Expert-answer costs are estimated from qualifications, typical problem time, and compensation evidence; Table 1 reports frontier costs by model and dataset.The table highlights the three lowest-cost entries in each dataset column.Source: §2.6.1 · p. 5, Table 1 · p. 6

3 EXPERIMENTS

Across five experiments, the paper compares language-model and expert cost-of-pass, tracks frontier changes over time, identifies model-family contributions, and evaluates inference-time techniques. The results show task-specific model advantages and limited economic gains from performance-oriented techniques.

3.1 EXPERIMENT SETUP

  • The study compares representative lightweight, large, and reasoning models, prioritizing earliest stable releases from the second half of 2024 through early 2025.Lightweight models are selected using a cost threshold of less than $1 per million input and output tokens.Source: §3.1 · p. 6
  • The evaluation covers basic quantitative, knowledge-based, and complex quantitative tasks, including Two Digit Addition, GSM8K, GPQA-Diamond, BBQ, MATH-500, and AIME-24.The datasets span arithmetic and grade-school problem solving, factual and bias-related questions, and challenging mathematical reasoning.Source: §3.1 · pp. 6–7

3.2 FRONTIER COST-OF-PASS WITH A SINGLE MODEL

  • Lightweight models had the lowest frontier cost-of-pass on basic quantitative tasks, large models on knowledge-based tasks, and expensive reasoning models reduced costs mainly on complex quantitative tasks.Using performance or cost alone tended to favor reasoning or lightweight models, respectively, without capturing their differing economic effects.Source: §3.2 · p. 7

3.3 TRACKING FRONTIER COST-OF-PASS WITH NEW RELEASES

  • MATH-500 frontier cost-of-pass halved approximately every 2.6 months, compared with 7.1 months for AIME-2024, between May 2024 and February 2025.Frontier cost-of-pass declined steadily for complex quantitative tasks; knowledge-based and basic quantitative tasks typically dropped sharply early and then plateaued.Source: §3.3 · pp. 7–8, Figure 2 · p. 7

3.4 ESSENTIALNESS OF MODEL FAMILIES: COUNTERFACTUAL FRONTIER COST-OF-PASS

  • Lightweight models helped reduce the frontier on basic quantitative tasks, large models were essential for knowledge-intensive tasks, and reasoning models advanced complex quantitative reasoning.Reasoning models also improved performance on GPQA-Diamond and GSM8K, where small reasoning models such as o3-mini helped.Source: §3.4 · p. 8, Figure 3 · p. 8

3.5 IMPACT OF INFERENCE TIME TECHNIQUES ON FRONTIER COST-OF-PASS

  • TALE-EP meaningfully reduced frontier cost-of-pass on more tasks, while majority voting offered no relative economic improvement and self-refinement yielded 24.9% improvement on GPQA-Diamond.The authors report that TALE-EP's benefits were uneven and that performance-oriented methods often had costs outweighing their accuracy gains.Source: §3.5 · p. 9, Table 2 · p. 9

4 RELATED WORKS

Related work examines model performance alongside inference resources, while cost-aware evaluation has used fixed budgets, heuristic scores, and non-monetary proxies. This paper positions its framework as a unified, monetary approach to connecting model performance and cost.

Framework contribution

  • The paper presents cost-of-pass as an interpretable monetary bridge from model performance to economic consequences, intended to assess model and technique viability without predefined budgets or proxy metrics.The authors describe it as a unified evaluation grounded in economic production principles.Source: §4 · pp. 9–10

Resource consumption and efficiency

  • Efficiency research covers monetary inference prices, token and energy costs, task-dependent performance, and possible diminishing returns from high expenditure.Reasoning methods and inference-time techniques can trade increased computational cost for potential accuracy gains.Source: §4 · p. 9

Cost-aware evaluation

  • Prior cost-aware evaluations have used fixed inference budgets, performance heuristics, or non-monetary measures such as conciseness, often tying conclusions to particular constraints or proxies.Related work also advocates real dollar costs and accounting for stochasticity, while benchmarking has noted diminishing returns from scaling inference computation.Source: §4 · p. 10

5 CONCLUSION

The paper concludes that cost-of-pass provides an economic framework for evaluating language models by combining performance and inference cost. Its evaluations identify different model-class roles and find that common inference-time techniques typically do not provide net economic benefits.

5 CONCLUSION

  • The framework combines inference cost and performance through cost-of-pass and frontier cost-of-pass measured against a human-expert baseline.Cost-of-pass measures expected cost per correct solution, while the frontier is the minimum achievable cost under the comparison.Source: §5 · p. 10
  • Retrospective and counterfactual evaluations identify lightweight models as primarily driving efficiency on basic tasks and reasoning models as essential for complex problem-solving.The conclusion also states that different model classes play distinct economic roles.Source: §5 · p. 10
  • The authors conclude that common inference-time techniques typically increase cost-of-pass without net economic benefits relative to progress from improving underlying models.They describe the framework as a principled basis for measuring model innovation in economic terms and guiding model selection.Source: §5 · p. 10

A DETAILS OF HUMAN EXPERT COST ESTIMATION

Human-expert costs are estimated per dataset using qualifications, hourly compensation, and typical solving time. The supporting estimates draw on dataset studies or related evidence when available, with task context used where direct data is absent.

A DETAILS OF HUMAN EXPERT COST ESTIMATION

  • Human-expert costs are estimated for each dataset using typical qualifications, hourly compensation, and time per problem, prioritizing evidence from dataset creation or associated studies.When direct evidence is absent, the method draws on related work or infers parameters from dataset context.Source: A DETAILS OF HUMAN EXPERT COST… · p. 15, Table 3 · p. 15
  • AIME estimates use about 12 minutes per problem and competition-math tutoring rates of $45–$100 per hour, while MATH-500 assumes roughly 12 minutes per problem and about $50 per hour.The MATH-500 estimate describes a typical competition-math tutoring range of $35–$60 per hour.Source: A DETAILS OF HUMAN EXPERT COST… · pp. 15–16
  • BBQ estimates use crowdworker pay of around $15 per hour and about 24 seconds per question, while GPQA-Diamond estimates reflect expert pay of about $100 per hour and at most two questions solved hourly.The GPQA-Diamond passage reports that skilled non-experts spent over 30–35 minutes per question on average.Source: A DETAILS OF HUMAN EXPERT COST… · p. 15
  • GSM8K estimates use about 3.7 minutes per question and tutor rates of $33–$55 per hour, while two-digit addition assumes roughly 2–3 seconds per problem and $15 per hour.The two-digit addition hourly rate is presented as an example within a broader $10–$20 range for basic math or data-entry work.Source: A DETAILS OF HUMAN EXPERT COST… · pp. 15–16

B DETAILS OF EVALUATION

The evaluation samples model outputs to estimate accuracy and runtime cost, defining per-attempt cost from input and output tokens and their prices. The expected-accuracy analysis averages results across sampled problem instances.

B DETAILS OF EVALUATION

  • Per-attempt cost sums input-token and output-token counts multiplied by their respective prices.Expert costs use upper-bound hourly rates to approximate expert accuracy as 1; proprietary models use original providers, while open-source models use TogetherAI.Source: B DETAILS OF EVALUATION · p. 16
  • Models are treated as inference pipelines, and comparisons include their prompts and decoding settings.The evaluation samples up to 128 problems per dataset and runs each model eight times, using specified sampling settings and default token limits.Source: B DETAILS OF EVALUATION · p. 16

C.1 EXPECTED ACCURACY AND INFERENCE COSTS

  • Expected accuracy and cost are estimated by averaging each model’s accuracy and cost across sampled problems; Tables 5 and 6 report those measures.The tables show accuracy and dollar cost per model and dataset, respectively; the authors note that each measure alone favors particular model families.Source: §C.1 · p. 17, Table 5 · p. 17

C.2 EVALUATION ON A REAL-WORLD DOMAIN

The real-world evaluation uses Tau-bench tasks and reports that all model families have merit, while frontier cost-of-pass continues to decay exponentially. Additional analyses examine gains by release, human-expert essentialness, and whether individual models’ contributions can be compensated by others.

C.2 EVALUATION ON A REAL-WORLD DOMAIN

  • All model families have merit on Tau-bench, its frontier cost-of-pass continues to decay exponentially, and no family is significantly essential for progress.The evaluation covers 16 airline and retail tasks, excludes DeepSeek-R1 under the protocol, and averages estimates over four independent trials; Table 9 reports the family essentialness analysis.Source: §C.2 · pp. 17–18, Table 9 · p. 19

C.3 RELATIVE GAIN PER MODEL RELEASE

  • Reasoning models show notable gains on complex quantitative tasks, while lightweight models show marked gains on basic tasks.The authors note that GPT-4o’s substantial improvement likely reflects its being the first model in their analysis relative to the human-expert baseline.Source: §C.3 · p. 18

C.4 ESSENTIALNESS OF HUMAN EXPERT BASELINE

  • Human experts remain fully essential for GSM8K, GPQA-Diamond, MATH-500, and AIME-2024, but are non-essential for BBQ and Two-Digit Addition.No task shows partial human-expert essentialness.Source: §C.4 · p. 19

C.5 ESSENTIALNESS OF SINGLE MODELS

  • Most individual models’ contributions to the frontier are largely compensable by the remaining models.Figure 5 shows relative frontier cost-of-pass improvement when a model is removed; family-level benefits still vary by task category.Source: §C.5 · pp. 19–20, Figure 5 · p. 20

C.6 REGIONAL ABLATION FOR HUMAN EXPERT COST ESTIMATION

Changing the human-expert cost estimate to India leaves the main interpretation intact, with a slight shift toward lightweight models. A bootstrap analysis largely mirrors the original findings, with one reported GPQA-Diamond ranking difference.

C.6 REGIONAL ABLATION FOR HUMAN EXPERT COST ESTIMATION

  • Using India-based expert costs leaves the main interpretation unchanged but slightly shifts preference toward lightweight models.The authors attribute the shift to lower expert costs making the framework more forgiving of errors; Table 11 repeats the Table 1 setup.Source: §C.6 · p. 20, Table 10 · p. 20, Table 11 · p. 21

C.7 CONFIDENCE INTERVALS FOR FRONTIER COST-OF-PASS ESTIMATES

  • In the bootstrap analysis, o1-mini has lower frontier cost-of-pass than Llama-3.1-405B on GPQA-Diamond, while the remaining findings closely mirror Table 1.The analysis uses 95% bootstrap percentile confidence intervals from 10,000 samples; the authors describe the results as indicating statistical significance.Source: §C.7 · pp. 20–21, Table 12 · p. 21

C.8 BENCHMARK HACKING: CHEAP RANDOM GUESSERS ON FORGIVING MULTIPLE-CHOICE TASKS

On forgiving multiple-choice tasks, the framework’s cost-of-pass can favor a very cheap random guesser over more capable models. The paper illustrates this benchmark-hacking risk and recommends excluding impractical baselines or strengthening evaluation and cost criteria.

C.8 BENCHMARK HACKING: CHEAP RANDOM GUESSERS ON FORGIVING MULTIPLE-CHOICE TASKS

  • On a k-option multiple-choice task, a format-following random guesser has expected accuracy approximately 1/k, so its cost-of-pass can be low when its cost is sufficiently small.The paper notes that this can undercut more capable models that reason about the problem but incur higher token costs or prices.Source: §C.8 · pp. 21–22
  • A cheap random guesser attains artificially low frontier cost-of-pass on the multiple-choice benchmarks BBQ and GPQA-Diamond.The adversarial baseline uses Llama-3.1-8B configured to output a random answer in the required format and appears to outperform genuinely competent systems on the frontier measure.Source: Table 13 · p. 22
  • The authors recommend excluding impractical baselines, using robustness-oriented dataset augmentation or stricter pass criteria, and penalizing failed attempts.Their example augmentation shuffles answer options independently and requires a consistent semantic answer across variants.Source: §C.8 · p. 22

D PRACTICAL IMPLICATIONS, LIMITATIONS, AND FUTURE DIRECTIONS

The section discusses ways to extend the framework’s cost and success models to more complex settings, while noting that expected values do not capture risk. It also identifies practical alternatives to dollar cost and binary success.

D.1 EXTENDING MODELING ACROSS COMPLEX AND DIVERSE DIMENSIONS

  • The cost model can add verification, failure, tool-use, orchestration, and amortized infrastructure costs beyond per-token API prices.These components can be represented by adding their unit costs and per-attempt quantities to the model.Source: §D.1 · p. 22
  • Success can be defined with stricter or more lenient criteria, including pass^k, pass@k, or minimum-recall conditions for classification.These alternatives address settings where reliability, robustness, partial correctness, or precision–recall trade-offs matter.Source: §D.1 · p. 22
  • For customer-support workflows, the model can incorporate tool fees, expert verification, failure losses, batching, and reliability-oriented success criteria.The example already accounts for tokens across multiple turns, but describes further extensions to represent economic objectives.Source: §D.1 · pp. 22–23
  • FLOPs, time, latency, or energy can replace dollars as the expenditure unit when they better fit an application.With parallel votes, majority voting may improve expected correctness while incurring roughly the latency of one vote.Source: §D.1 · p. 23

D.2 INTERPRETABILITY OF OUR FRAMEWORK

  • Cost-of-pass improvements can come from lower unit prices, reduced resource use, or increased success probability.Examples include cheaper tokens, prompt compaction, dynamic budgets, and improved prompting or training.Source: §D.2 · p. 23
  • Expected cost-of-pass does not capture variance, so strategies with equal expected values can entail different risks.The section suggests variance measures, risk-adjusted objectives, or median and worst-case values as possible extensions.Source: §D.2 · p. 23

D.3 LIMITATIONS

The evaluations and framework have scope limits involving cost and success definitions, model and benchmark selection, and the human-expert comparison. The authors describe extensions that could address richer costs, utilities, risk, and strategy sets.

D.3 LIMITATIONS

  • Core results omit indirect and context-specific costs, including verification overhead, wait times, retries, and tool-call charges.The framework can represent such terms, but the experiments use simpler cost and success formulations.Source: §D.3 · p. 23
  • Binary success does not directly model continuous or composite outcomes, for which the authors propose cost per unit of utility as future work.A utility function could distinguish outcome quality and encode asymmetric values for errors such as false positives and false negatives.Source: §D.3 · pp. 23–24
  • Provider-level pricing and performance can vary, so treating each provider–model pair separately may make comparisons more robust.Source: §D.3 · p. 24
  • Using one instruction and fixed sampling settings aids comparability, but results may be sensitive to those choices.The authors suggest prompt ensembles and decoding sweeps for future sensitivity analysis.Source: §D.3 · p. 24
  • Public benchmarks may overlap with training data, so results may optimistically estimate cost-effectiveness on familiar problems rather than out-of-distribution generalization.The authors advise accounting for potential training–test overlap, especially in high-stakes settings.Source: §D.3 · p. 24
  • Budget and coverage constraints limited evaluation to representative releases, while model-family groupings and the focus on common user-facing models are choices rather than exhaustive categories.The paper proposes broader release coverage and alternative categorizations for future work.Source: §D.3 · p. 24
  • The human baseline assumes near-perfect expert success, and evaluations omit hybrid human–AI workflows; estimating their costs requires human-subject studies.Source: §D.3 · pp. 24–25
  • The inference-technique findings use a risk-neutral, dollar-cost setup, though richer risk or reliability requirements could favor heavier techniques.The framework can accommodate refined costs, richer success metrics, and variability terms, while the current analysis uses per-token prices and binary pass/fail.Source: §D.3 · p. 25
Loading 2504.13359v2…