PaperlayerView exact revision on arXiv

Source-linked AI summary

The Price of Intelligence: A Quality-Adjusted Price Index for AI Services

Louis Yiven Zhu

arXiv:2608.29843v1econ.GNcs.LGq-fin.EC

TL;DR

AI inference price estimates depend on whether they adjust for model quality and count prices per token or per completed task. This paper builds and audits a benchmark-based capability measure for quality-adjusted indices; the audit finds that a ranking-preserving benchmark exclusion shifts the index substantially.

  • Problem

    AI inference price estimates disagree substantially, although price series feed measures of provider competition, market concentration, and sector productivity.

    Source: §1 · p. 1

  • Method

    The paper joins public posted prices to benchmark scores, estimates latent capability from benchmark response patterns, audits that measure, and constructs price indices in two units.

    Source: §1 · p. 2, §4 · p. 6

  • Results

    A 0.998 capability-ranking correlation survives flagged-benchmark exclusion, yet the index shifts by 0.49 log points a year.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The paper concludes that benchmark-based adjustment is too large to forgo, but economic statistics require instrument-grade care and deliberate unit choices.

    Source: §10 · p. 18

    The indices use posted list prices on one first-party channel per provider; because transaction prices are lower, measured declines understate true declines.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Posted prices for AI inference have fallen steadily since 2024, yet the measured speed of that fall depends almost entirely on the method of measurement. This paper constructs quality-adjusted price indices for the AI inference market from public data. The panel assembles 21,024 posted-price observations across 3,208 models and 86 providers and joins them to 4,605 benchmark scores through a latent quality index estimated from benchmark response patterns, so the quality ladder of the hedonic tradition is built here from evaluations in place of product characteristics. Measured by the matched-model methods that statistical agencies apply to software, inference prices fell at 0.10 log points a year. The quality-adjusted index fell at 0.73, so 87% of the decline is invisible to current methods, with direct consequences for measured competition, concentration and productivity in this market. Counted per completed task, moreover, the buyer's price stopped falling. Reasoning models raised token consumption faster than token prices fell, and the seller's and buyer's prices accordingly diverged. A pre-registered validity audit disciplines the quality measure and yields the sharpest result. Excluding contamination-flagged benchmarks leaves model rankings intact at 0.998 yet moves the index by 0.49 log points a year, so the leaderboard-stability arguments standard in AI evaluation offer no defence of economic statistics built on benchmarks. Prices, quality and the audit are fully reproducible from public sources at zero cost.

1 Introduction

AI inference price estimates diverge according to whether they account for model quality and whether prices are counted per token or per completed task. The paper builds and audits a benchmark-based capability measure to compare these approaches.

1 Introduction

  • 0.73 versus 0.10 log points a year: quality-adjusted prices fall faster than matched-model prices, and quality adjustment captures 87% of the decline.The matched-model rate follows the software-measurement approach described in the introduction.Source: §1 · pp. 1, 3
  • Per completed task, prices stop falling over the measurable window, unlike the decline measured per token.The introduction identifies this as a distinct answer to the price-measurement question.Source: §1 · p. 1
  • The wedge arises because new models bring quality at old prices, a margin matched-model methods miss; benchmarks could supply the missing quality characteristic.Benchmarks also saturate, leak into training, and draw scores from heterogeneous and sometimes self-interested sources.Source: §1 · p. 2
  • The paper assembles public price and benchmark data, estimates and audits latent capability, then estimates four price indices in two units.Its methods and sensitivities are pre-registered, and the analysis is reproducible at zero cost.Source: §1 · p. 2
  • A 0.998 ranking correlation survives flagged-benchmark exclusion, yet the quality-adjusted index shifts by 0.49 log points a year.The paper calls this its central result; two of the audit’s four inferences do not pass.Source: §1 · pp. 2–3

2 Related work

The paper joins hedonic price measurement, psychometric validity auditing, and AI evaluation research to construct an audited capability measure for price indices. Its novelty is the economic application and audit, rather than a new model class.

2 Related work

  • Hedonic adjustment supplies the framework for quality change, while ICT-deflator mismeasurement provides a template for examining its stakes.Modern practice combines hedonic estimation with expenditure-weighted superlative formulas, but public AI data lack expenditures, bounding the paper’s weighting.Source: §2 · p. 3
  • Unlike Bajari et al. (2025), this paper validates and compresses existing quality characteristics; hedonic theory bounds its index as a price schedule, not welfare.The interpretive boundary is tied to Rosen (1974) and Pakes (2003).Source: §2 · p. 3
  • Psychometrics supplies validity arguments, equating, plausible values, and differential item functioning; AI evaluation research supplies evidence on contamination and saturation.The paper applies these established tools to economic measurement rather than claiming novelty in the model class.Source: §2 · pp. 3–4
  • Epoch’s Capabilities Index uses the same score corpus, but this paper propagates posterior draws downstream and subjects its estimate to an adversarial audit.Efficiency-frontier measures of fixed capability levels differ from a deflator, which aggregates across the quality spectrum and carries uncertainty and a validity argument.Source: §2 · p. 4

3 Data

The data section describes a provenance-rich public panel of posted prices and benchmark scores, then defines the boundaries of its price and task measures. It also reports the public release of the data, code, and reproducibility materials.

3.1 Sources and assembly

  • Benchmark scores and model metadata come from Epoch AI; prices and token consumption come from Artificial Analysis, with archived pages enabling recovery of hidden pricing history.OpenRouter, LMArena, and provider pricing pages provide cross-checks or audit inputs.Source: §3.1 · p. 4
  • 21,024 observed price rows cover 3,208 models, 86 providers, and 31 months, alongside 4,605 scored model–benchmark cells across 64 benchmarks.Each price row retains its source URL and capture timestamp; the panel also records 601 explained in-window gaps without interpolation.Source: §3.1 · p. 4
  • Score provenance varies: 23.3% of cells are independently evaluated, 33.9% come from third-party leaderboards, 26.0% are developer-reported, and 16.8% unattributable.Developer-reported scores concentrate in older benchmarks that anchor the early capability scale.Source: §3.1 · p. 5

3.2 What a posted price is and is not

  • List prices use one named first-party channel per provider; transaction prices are lower, so the measured declines understate true price declines.A single launch showed three simultaneous channel prices, while the task series starts later because consumption data are available over a shorter window.Source: §3.2 · pp. 5–6

3.3 Release

  • Data, code, analysis artifacts, hashes, provenance ledgers, and the pre-registration are publicly released with a record of deviations.The data use CC BY 4.0 and the code MIT licensing; the release is archived under DOI 10.5281/zenodo.22177190.Source: §3.3 · p. 6

4 Measuring capability

The paper builds a hedonic quality index from benchmark response patterns, linking models across benchmark generations and carrying capability-estimation uncertainty into downstream results. External checks offer some support, while the authors describe their limits.

4 Measuring capability

  • Each model receives a latent capability estimate with posterior uncertainty, avoiding the coverage, retirement, and saturation problems of relying on one benchmark.The model jointly estimates benchmark difficulty and discrimination; downstream analysis treats capability as data with error, not truth.Source: §4 · p. 6
  • The continuous response model fits 4,605 scored cells across 782 models and 64 benchmarks, with a shared residual standard deviation estimated at 0.449.The shared residual restriction is required because the beta likelihood and benchmark-specific residual variances produce degeneracies in these data.Source: §4 · p. 6
  • Two-stage fixed-parameter equating links benchmark generations through anchor parameters, making the linkage explicit while allowing anchor defects to propagate.GPQA Diamond sets the scale, and anchor benchmarks are fitted first before their parameters are held fixed during estimation of the remaining parameters.Source: §4 · pp. 6–7
  • Twenty posterior draws per model carry capability uncertainty through downstream quantities, while month coefficients remain invariant to time-constant affine rescaling of capability.Drift in equating over time would be fatal to the index, which is why the audit checks differential functioning.Source: §4 · p. 7
  • Capability correlates with release date at Spearman 0.880 across 757 dated models, a weak validity check; within-benchmark item-model comparisons reach 0.7 or better on three of five benchmarks.The item-level comparison is confined within benchmarks because readable logs share too few models across benchmarks to support equating.Source: §4 · p. 7

5 Auditing the capability measure

The audit tests whether benchmark scores support the capability index through scoring, generalisation, extrapolation, and decision inferences. Its mixed outcomes identify where the capability characteristic is thinnest and how the index should be used.

5.1 Scoring: differential functioning and untestable cohorts

  • Six of nine testable anchors flag differential functioning after a quadratic capability control, while seven anchors lack a pre-release comparison group for the registered test.Calendar-cohort flags are operative scoring evidence, not proof of contamination; the six flagged benchmarks form the sensitivity analysis exclusion list.Source: §5.1 · p. 8

5.2 Generalisation: dimensionality and saturation

  • The data do not establish whether capability is one dimension or several: median anchor-pair rank agreement is 0.809, but weak pairs involve GSM8K and older knowledge benchmarks.Three of ten anchors saturate in-window under the ceiling-proximity measure; the count ranges from one at ε = 0.02 to six at ε = 0.10, while MATH Level 5's status is unchanged.Source: §5.2 · p. 9

5.3 Extrapolation: external criteria

  • θ tracks LMArena preference ratings at Spearman −0.700 across 69 models and Epoch’s capability index at 0.930 across 504, though the latter shares benchmark inputs.Release date also correlates at 0.880 across 757 dated models, but the paper calls this a weak test of validity.Source: §5.3 · p. 9

5.4 Decision: where the index can be trusted

  • Posterior standard errors fall toward the frontier, from a median 0.187 in the lowest capability quintile to 0.084 in the highest, because coverage explains much of the pattern.Conditioning on scored cells raises explained variance from 0.126 to 0.690; the authors caution that their specification cannot express the saturation channel.Source: §5.4 · p. 10

5.5 Dispositions

  • The audit’s four dispositions are scoring fails, generalisation remains unestablished, extrapolation passes, and the decision inference passes with its expected sign reversed.These findings do not disqualify the index; they locate the capability measure’s weaknesses and redirect the reliability warning toward the back catalogue.Source: §5.5 · p. 10

6 From capability to a price index

The section builds a rolling-window hedonic index from benchmark-based capability, using entrant-aware estimation and movement-level uncertainty. It also states the index’s interpretive boundary.

6 From capability to a price index

  • The validity chain reports one failed inference, one undetermined inference, and two passes; its robustness check also fails.The scoring failure supplies the exclusion list; the extrapolation pass rules out gross failure without licensing confidence.Source: §6 · p. 11, Table 1 · p. 11
  • The rolling-window hedonic E3b admits new models at entry and captures a capability premium that drifts from 0.789 to 0.452 across thirteen-month windows.The pooled hedonic imposes one premium; a twenty-five-month window flattens the drift.Source: §6 · pp. 11–12
  • Month coefficients δt trace pure price change at constant capability, and exponentiating them yields the index.The per-task unit uses p × n(m), where n(m) is tokens consumed to complete a benchmark task; consumption is measured once per model, so within-model drift is invisible.Source: §6 · p. 12
  • Movement-level inference forms contrasts before applying Rubin’s rules across twenty plausible values; treating coefficients as independent inflates median movement standard errors more than sevenfold.The spliced movements use the full cluster-robust covariance.Source: §6 · pp. 12–13
  • The joined sample contains 6,158 priced observations across 500 capability-scored models, and identification relies on comparable-capability price variation within each window.Indices are unweighted and recover the equilibrium price schedule, not welfare; usage weighting would likely make the index fall faster.Source: §6 · p. 13

7 Results

Matched-model indices capture little of the quality-adjusted decline, while the buyer-side task comparison and benchmark audit qualify how the price trend should be read.

7 Results

  • 87% of the quality-adjusted decline is invisible to current methods: matched-model prices fell 0.097 log points a year versus 0.728 for the quality-adjusted index, a 0.630-point wedge.The matched-period Jevons rate was 0.098; the wedge interval was [0.222, 1.004].Source: §7 · p. 13
  • Adding provider fixed effects moves the headline rate from 0.728 to 0.765; holding the seller fixed makes the measured decline slightly faster, so provider composition contributed nothing to the fall and if anything damped it.Across 7-, 13-, and 25-month windows, the quality-adjusted rate runs from 0.412 to 0.765, with each window on its own stated base.Source: §7 · pp. 13–14
  • +0.98 log points a year separates the per-task and per-token rates, with interval [+0.02, 1.95]; the task rate’s own sign is not established.On the selected window, the token-denominated rate was −0.506 and the task-denominated bootstrap point estimate was +0.470; the thinner task series and discarded sparse bootstrap resamples qualify the comparison.Source: §7 · pp. 14–15, Table 2 · p. 15
  • 0.998 correlation between capability rankings coexists with a 0.487-log-point annual index shift after excluding flagged benchmarks.The index’s dependence on those benchmarks is conditional: contamination and cohort differences are both possible explanations for the flags.Source: §7 · pp. 14–15

8 Implications for economic measurement

The paper illustrates how deflator choices change measured real growth, while emphasizing that the task-based result and the broader policy application have important qualifications. Benchmark adjustment is feasible, but agencies must make deliberate choices about validity, units, and market coverage.

8.1 An illustrative deflator bound

  • 0.63 log points a year is the illustrative real-growth wedge between matched-model and quality-adjusted deflation; the nominal base, list-price foundation, and panel-composition assumption limit its scope.The nominal base is a survey-based estimate of enterprise model-API spending; the authors say the exercise is illustrative and the wedge understates mismeasurement.Source: §8.1 · p. 15
  • Per completed task, the price level stopped falling over the measurable window, so task- and token-based deflation give real-growth estimates of opposite tenor.The per-task bootstrap wedge is 0.59 with interval [-0.60, 1.73] over five months and is not established.Source: §8.1 · pp. 15–16, Figure 3 · p. 16

8.2 What this says to current proposals

  • Benchmark-based adjustment is feasible with public data and standard hedonic methods, but requires a latent capability measure, metric-structure auditing, and a defended choice of unit.The authors also warn that capability and task-unit measurement are weakest in the sparsely evaluated back catalogue.Source: §8.2 · pp. 16–17
  • 0.998 rank correlation across 496 models coexists with a 0.487-per-year index movement after excluding six flagged anchors, showing that ranking robustness does not settle index sensitivity.The mean absolute level difference reaches 27.2.Source: Figure 4 · p. 17, §8.2 · p. 17

8.3 The price of intelligence, so far

  • The seller’s token price fell fast, its capability-adjusted price fell over seven times as fast, and the buyer’s price per completed task stopped falling; archival benchmark integrity remains a measurement concern.The authors call versioned benchmarks with exposure dates relative to model training a public good for economic measurement; they say no one is currently producing it.Source: §8.3 · pp. 17–18

9 Limitations

The findings are bounded by the available price channel, uneven task-series coverage, and unresolved measurement questions; the paper also concerns one product market in one period.

Limitations

  • The indices measure posted list prices on one channel, not transaction prices, and the authors argue that true price declines are faster.Discounts and agreements are not publicly observable, and their gap from list prices widened nonuniformly over the sample.Source: §9 · p. 18
  • The task series spans seventeen months, its wedge is not established, and the reported divergence uses the seven-month window selected by the pre-stated rule.At the 13-month window, the per-task series falls faster than the token series.Source: §9 · p. 18
  • The weighting direction is argued rather than estimated, the exclusion list measures dependence rather than contamination, and the generalisation inference remains open.The early equating relies on developer-reported saturated benchmarks that have no replacement in the analysis.Source: §9 · p. 18
  • The results cover one product market in one period, and three pre-registered elements changed during the work and were logged and disclosed.Source: §9 · p. 18

10 Conclusion

The conclusion argues that benchmark-based quality adjustment matters for AI inference prices, but requires careful measurement of capability, validity, and units. The accompanying appendix records the methods and data procedures behind the reported statistics.

10 Conclusion

  • A matched-model index misses 87% of the quality-adjusted decline, while a ranking-preserving exclusion moves the index by half a log point a year and seller and buyer prices diverge.The conclusion calls for instrument-grade care, including estimated capability, metric-level validity audits, deliberate unit choices, and reproducible statistics.Source: §10 · p. 18

A How to read this appendix

  • The appendix links each number to a named field in a committed artifact with recorded input hashes, enabling readers to trace statements to the code that produced them.Source: A How to read this appendix · p. 21

B Mathematical basis

  • The response model uses a shared residual scale because beta and per-benchmark-variance likelihoods degenerate on these data; anchor parameters are fixed before estimating remaining benchmarks and model capability.The two-stage equating makes the linkage explicit but allows anchor defects to propagate, motivating the anchor audit.Source: B Mathematical basis · pp. 22–23, Table 3 · p. 22
  • Downstream statistics use 20 plausible capability values and Rubin combination, propagating capability uncertainty; price sampling supplies a median 4.9% of movement variance.Movement uncertainty retains the covariance between monthly estimates; ignoring it inflates median movement standard errors by a factor of 7.6 and produced impossible negative index-level bounds.Source: B Mathematical basis · p. 23, Table 5 · p. 24

C Procedure boxes

  • The appendix specifies the build order, matching tiers, and freeze guards used to check the paper’s reproducibility claims.Source: C Procedure boxes · p. 25

D Panel construction and provenance

  • The price panel combines text and embedded-state archival recovery; hand checks and adjudication document verification limits and the treatment of divergent listings.The structural pass recovered 13,829 observations absent from visible text; 40 recent, current-page observations matched their source URLs without correction, while a 1,051-row random sample is released for inspection.Source: D Panel construction and provenance · p. 25, Table 7 · p. 27, Table 8 · p. 27
  • Figure 5 maps provider-by-month coverage, with darker cells marking months recovered from embedded application state and lighter cells months with rendered text only.Coverage thins toward the market’s beginning, so the panel starts under the pre-registered three-provider rule and earlier months appear only in the descriptive prologue.Source: Figure 5 · p. 26
  • The appendix defines gaps without interpolation and treats retired endpoints as exits, with the in-window count distinguished from neighbouring definitions.Source: Table 9 · p. 28

E Benchmark battery

The benchmark battery records scores, provenance, dates, saturation, and anchor roles. A pre-stated rule selects anchors that link model eras while retaining variation among recent models.

E Benchmark battery

  • The battery contains 64 benchmarks, with table fields recording anchor role, scored-model counts, score provenance, release dates, and saturation dates.The roster includes benchmarks with differing model counts and provenance sources, including Epoch, third-party leaderboards, lab reports, and unattributed scores.Source: Table 10 · p. 28, E Benchmark battery · p. 29

F Anchor selection

  • Ten benchmarks qualify as anchors by having at least ten scored cells on each side of 1 July 2024 and post-boundary score standard deviation of at least 0.10.The first condition links eras; the second excludes benchmarks too saturated to distinguish recent models. Raising the count threshold to 20 leaves four anchors and moves the headline rate 0.020 log points a year slower.Source: F Anchor selection · p. 29

G Audit evidence

The audit finds important limits in the benchmark evidence: the primary cohort test is untestable for most anchors, while item-level fits and cross-anchor rankings show qualified agreement. Item information also peaks below each benchmark’s 90th capability percentile, and the estimation-detail diagnostic reports substantial capability overlap between adjacent months.

G Audit evidence

  • Weak reliable cross-anchor ranking pairs all involve GSM8K against older knowledge anchors, suggestive of more than one dimension but not conclusive.The negative GSM8K pair with a recent mathematics benchmark uses only eleven shared models and is not reliable evidence.Source: Figure 6 · p. 30
  • The primary pre-and-post-release test is untestable for seven of ten anchors, while the calendar-year design flags six of nine testable anchors after the ceiling-curvature control.The flags form the contamination-exclusion list whose index consequences Section 7 quantifies.Source: Table 13 · p. 31
  • Three of five readable benchmarks have rank correlation of 0.7 or better between item-level fits and the continuous-response capability estimate.The two failures involve one benchmark with raw accuracy tracking capability at 0.21 within a narrow ability band and another with eleven usable items.Source: Table 15 · p. 32
  • Every item-information curve peaks below its benchmark’s own p90, making each benchmark least informative about models at the top of its range.The curves demonstrate the ceiling mechanism, but only the solid-curve benchmark is dated by the ceiling-proximity instrument.Source: Figure 7 · p. 33
  • All four reported wedges round to 0.630; three equal 0.6305, while the fourth decimal separates only the per-token E1 wedge.The per-task E1 and E2 annualised rates match to four decimals, making their equal wedges a construction rather than a coincidence.Source: G Audit evidence · p. 34

H Index estimation detail

  • Adjacent-month capability-range overlap never falls below 0.898 per token or 0.833 per task, above the identification floor of one half.Figure 8 presents this as the identification diagnostic.Source: H Index estimation detail · p. 34

I Robustness, registered and delivered

The robustness materials distinguish registered checks from added analyses and emphasize that rolling-window levels need their own bases. They document the anchor-selection rule, three logged deviations from registration, and a reproducible public-data pipeline.

I Robustness, registered and delivered

  • Rolling-window levels are not comparable across rows because each window has its own base; the 13-month series starts later yet ends lower because it covers the fastest price-fall months.Quote rates with their ranges and levels only with their base; the capability-premium drift motivates preferring rolling windows to a pooled estimate.Source: I Robustness, registered and delivered · p. 35, Table 17 · p. 35

J Deviations from the pre-registration

K Reproducibility and data provenance

  • One command rebuilds the public-data pipeline, and embedded input hashes, pinned PDF builds, and timestamped provenance ledgers support reproducibility.The ledgers connect observations to source URLs and timestamps; the pre-registration and public deviation log are linked.Source: K Reproducibility and data provenance · p. 36
Loading 2608.29843v1…