Source-linked AI summary
On the estimation and validity of AI time horizons---a statistical look at the METR plot
Drew T. Nguyen, William Fithian
TL;DR
The paper asks what the METR time-horizon plot estimates and whether horizons are useful measures of AI capability. It refits the estimates and proposes diagnostic plots; its proposed estimates dominate the chosen baseline on reported scoring-rule metrics.
Problem
The paper asks what quantity the METR time-horizon plot estimates and whether time horizons are useful measures of AI capability.
Source: §2.1 · p. 4Method
The authors fit new time-horizon models, use Model 2 to estimate the human-time–difficulty relationship, and propose plots to diagnose construct validity.
Source: §1 · p. 3Results
The proposed point estimates dominate the chosen baseline on all reported scoring-rule metrics.
Source: §2.5 · p. 9Takeaways & Limitations
The authors present diagnostics as a lens for interpreting time horizons alongside improved estimates.
Source: §1 · p. 3The diagnostics assume affirmative answers to the first three construct-validity questions and address only predictiveness and comparability.
Source: §3.1 · p. 11
Abstract
from arXiv · showhide
METR's 50\% time horizon measures the human completion time of software tasks that an AI solves with 50\% probability, allowing AI capabilities to be expressed in interpretable units. On 228 tasks and 26 AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the AI difficulty of a task depends linearly on the log of human time. Our fitted spline can be interpreted as a function that \emph{converts} human time to AI difficulty; it is nearly flat in a region from 2--30 min but close to linear elsewhere. Hence, a time-horizon jump from 3 min to 30 min is much easier than one from 30 min to 5 hours despite the same multiplier of $10 \times$. Overall, we contribute time-horizon point estimates that perform better under a cross-validated suite of proper scoring rules, as well as diagnostic plots for assessing time horizons' construct validity. We suggest that time horizons be interpreted together with the diagnostic plots, especially as new time-horizon-based benchmarks are proposed or existing ones grow to include longer tasks.
1 Introduction
The paper reanalyzes METR’s time-horizon estimates and assesses whether they provide a useful measure of AI capability. Its Model 2 performs better than the baseline under scoring-rule evaluations, while diagnostics reveal that human time does not map uniformly to AI task difficulty.
1 Introduction
- A time horizon is the human completion time of tasks an AI solves autonomously with 50% probability, expressing capability in interpretable units tied to task duration.The paper treats the plotted value as a statistical estimate that can be critiqued and improved, not a direct physical measurement.Source: §1 · pp. 1, 3
- Model 2 predicts success better overall than the baseline under scoring-rule evaluations, suggesting its conditional success plot better summarizes the data.The paper also reports that its proposed estimates improve on held-out data.Source: §1 · pp. 2–3
- The authors say prior critiques did not, to their knowledge, both claim a better estimator and examine when time horizons are substantively useful.They frame the work as a constructive critique that improves estimates and helps interpret them.Source: §1 · p. 3
- The authors propose time-to-difficulty conversion and conditional success trajectory plots to diagnose construct validity using Model 2.Model 2 estimates the relationship between a task’s human time and its latent difficulty.Source: §1 · p. 3
- AI difficulty is nearly flat across human times of 2–30 minutes, so a time-horizon jump can mean different things in that region than elsewhere.The authors suggest that a jump from 4 to 15 minutes may be quite small.Source: §1 · p. 3
1.1 Prior work
- The time-horizon approach anchors an IRT-based ability measure to an external scale, while the Epoch Capabilities Index combines benchmarks at the cost of less interpretability.The Lexile Framework is cited as an earlier example of anchoring an ability measure to an interpretable quantity.Source: §1.1 · p. 4
2 The time-horizon estimation problem
This section defines AI q-time horizons and describes the task data, model specifications, and cross-validated evaluation used to estimate them. It also reports where metric gains were concentrated: estimates for AIs with 2–15-minute horizons, especially six affected by tasks in the 2–30-minute flat region.
2.1 The statistical framework
- A q-time horizon is the human time of tasks an AI solves with probability q, obtained by inverting its strictly decreasing success curve; AI release dates are used only for plotting.The framework notes that misspecification can induce dependence among repeated runs for the same AI-task pair.Source: §2.1 · p. 4
2.2 Implementing the framework
- METR’s dataset contains 228 software-relevant tasks across 79 task families; human times use geometric means of successful engineer attempts, with expert values substituted for 29% of tasks.AIs attempted tasks autonomously in repeated runs, and their completion times were not used.Source: §2.2 · p. 5
2.3 Model specifications
- They use a specification with sqrt-family weighting, not the logistic regression specification, as their performance baseline; Model 1 replaces linear dependence on log2(t) with a shared monotone function.Assuming independent runs under the baseline generative model and correct fitted pij = pj(Ti), about 60% of runsets would be unanimous, versus 83% observed; the data exhibit substantial within-runset dependence.Source: §2.3 · pp. 6–7
2.4 Evaluation with proper scoring rules
- Evaluation uses proper scoring rules on success curves and time horizons; Murphy diagrams show score behavior across q, and dominance at every q implies dominance under every proper score.The smoothed elementary score directly measures time-horizon estimate quality.Source: §2.4 · p. 8
2.4.1 Evaluation weights
- The equal-task, equal-family, and sqrt-family weightings target different task distributions; sqrt-family weighting is METR’s intermediate scheme for fitting and evaluation.All schemes weight AIs uniformly while adjusting for unequal runs and, depending on the scheme, unequal family sizes.Source: §2.4.1 · p. 9
2.4.2 A suite of proper scoring rule metrics
- The evaluation suite crosses four scores with three weighting schemes, using five-fold cross-validation on the 79 task families.The scores are marginal log, Brier, and smoothed elementary binary scores at q=0.5 and q=0.8.Source: §2.4.2 · p. 9
2.5 Fit results and diagnostics
- Both proposed models’ point estimates dominate the shared-slope baseline on all 12 proper scoring metrics; that baseline beats METR’s logistic model on all but one.Figure 2 shows both proposed models ahead of the baseline at every displayed metric; the exception for the baseline versus METR’s model is equal-task smoothed elementary score at q=0.5.Source: §2.5 · p. 9, Figure 2 · p. 9
- For q > 0.4, the baseline is nearly dominated in Murphy diagrams, while the proposed fits bend through a roughly flat 2–30-minute region and alter some short-horizon estimates beyond METR’s 95% intervals.The in-sample diagnostics cover six AIs with horizons of 2–15 minutes; Figure 1(d) indicates tasks in the flat region have roughly equal difficulty.Source: §2.5 · p. 9
- A large share of metric gains came from improved estimates for AIs with 2–15-minute horizons; six such AIs were most affected by tasks in the 2–30-minute flat region.Figure 4 compares in-sample fitted solve probabilities with observed frequencies for those six AIs.Source: §2.5 · pp. 9–10, Figure 4 · p. 10
3 Diagnostics for construct validity
Time-horizon construct validity depends on how human time relates to AI task difficulty. The section proposes diagnostic plots to assess predictiveness and comparability, and considers Rasch abilities as a direct capability measure.
3.1 Construct validity for capability anchoring
- On METR’s software-task suite, human time predicts AI success strongly overall but is locally weak at 2–30 minutes, raising a comparability issue; the diagnostics address only predictiveness and comparability.The authors assume affirmative answers to the checklist’s first three construct-validity questions.Source: §3.1 · pp. 11–12
3.2 The time-to-difficulty conversion plot
- The conversion plot’s fitted spline changes shape, suggesting that time-horizon jumps in the 2–30-minute region should be interpreted differently from jumps elsewhere.The section treats predictiveness by comparing the fitted shape with an SNR = 1 reference line.Source: §3.2 · p. 12
3.3 Conditional success trajectories
- Curve spread reflects predictiveness, while variation between spreads suggests comparability issues; in Figure 1(f), the 4-to-15-minute horizon difference is much smaller for every conditional probability than other jumps.Figure 1(b)'s reasonably spread-out curves suggest good predictiveness, but its baseline model cannot assess comparability due to its functional form.Source: §3.3 · p. 13
3.4 Rasch abilities and the METR plot
- Model 2’s affine-rescaled Rasch abilities nearly match METR’s estimates more closely than its own time horizons, but this coincidence is not guaranteed and depends on spline shape.The models’ abilities directly target the construct of AI capability.Source: §3.4 · p. 13
4 Discussion: the future of time horizons
The discussion cautions that extending time horizons to longer tasks may reveal new nonlinearities in the human-time-to-difficulty relationship. It also compares Model 2’s horizons with METR’s and illustrates a hypothetical conversion beyond one day.
4 Discussion: the future of time horizons
- If tasks extend beyond the current 30-hour range, the conversion may become unusually flat or steep, which linear-model checks may miss.The authors say the future curve could resemble the hypothetical Figure 6.Source: §4 · p. 13
- For robotics and other applications beyond software engineering, human time may not be the best task annotation for capability anchoring.Source: §4 · p. 13
- Model 2’s rescaled Rasch abilities nearly match METR’s horizons, while its own horizons disagree most in the shaded 2–30-minute band.Figure 5 contrasts the two Model 2 measures against METR’s per-AI horizons.Source: Figure 5 · p. 14
- Figure 6 depicts one possible continuation of the time-to-difficulty conversion after one day, rather than an observed extension.The plot uses Figure 1(d) up to one day and presents a hypothetical future beyond it.Source: Figure 6 · p. 14
A.1 Per-AI evaluations
Per-AI evaluations show that Model 2’s point-estimate advantage is concentrated in the middle of the capability range, although most individual intervals are too wide to distinguish from zero. The in-sample shape misfit of the log-linear baseline is most evident for mid-strength AIs.
A.1 Per-AI evaluations
- Figure 7 reports Model 1 minus baseline paired held-out score differences for all 26 AIs, with values below zero favoring Model 1.Rows show score types and columns evaluation weightings; orange marks six AIs with 2–15-minute horizons.Source: Figure 7 · p. 16
- Model 2’s point-estimate advantage is concentrated in mid-capability AIs, but most individual-AI intervals remain too wide to separate from zero.The pooled comparison in Figure 2 is where the evidence accumulates.Source: Figure 8 · p. 17
- The log-linear baseline’s shape misfit is concentrated in mid-strength GPT-4-era AIs; curves differ mainly in the tail for weakest AIs and nearly coincide for strongest AIs.Source: Figure 9 · p. 18