PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models

Dewi Gould, Francis Rhys Ward, Anders Cairns Woodruff, Rauno Arike, Josh Hills, Alex Serrano, Ida Caspary, Jason Ross Brown, Jo J. Jiao, Patrick Leask, Twm Stone, Ram Potham, Ionut Gabriel Stan, Harry Mayne, Simeon Hellsten, Shubhorup Biswas, Ariana Azarbal, William L. Anderson, Elle Najt, Ryan Greenblatt, Julian Stastny

arXiv:2606.07157v4cs.AI

TL;DR

The paper addresses whether frontier models can reason substantially without exposing chain-of-thought, which matters for monitoring their reasoning. It measures no-CoT capability across 43 benchmarks using human-time and reasoning-token horizons, finding roughly annual growth and projecting substantially longer horizons by 2030.

  • Problem

    The paper asks how much reasoning frontier models can perform without externalizing CoT and how that latent capacity changes over time, because such reasoning may be harder to monitor.

  • Method

    The authors evaluate 14 models across 43 benchmarks and estimate 50% task-completion time horizons plus reasoning-token horizons using calibrated human times, an o3-mini anchor, and hierarchical bootstrap uncertainty.

  • Results

    No-CoT time horizons doubled every 373 days over six years, reaching over 3 minutes for GPT-5.5, while median projections exceed 7 minutes by 2028 and 25 minutes by 2030.

  • Takeaways & Limitations

    Frontier developers should track no-CoT latent reasoning abilities as models advance.

  • Takeaways & Limitations

    The estimates depend on elicitation strategies and cannot perfectly distinguish latent reasoning from memorisation or heuristic pattern-matching.

Abstract

from arXiv · show

Many efforts to ensure frontier AI models are safe rely on monitoring their chain-of-thought (CoT) reasoning. If models become able to perform sufficiently complex reasoning internally, without explicit thinking tokens, this would undermine such oversight. We measure how well frontier models reason without CoT across a suite of over 30,000 questions spanning 43 benchmarks in domains including math, coding, puzzles, causality, theory-of-mind, and strategic reasoning. To compare models against humans, we estimate the $50\%$-task-completion time horizon (TH): the human time required for tasks a model completes with $50\%$ success rate. We complement this with a $50\%$ reasoning token horizon: the minimum number of o3-mini reasoning tokens needed for tasks a model solves with $50\%$ success rate. We find that the no-CoT $50\%$ TH of frontier models has been doubling roughly every year over the past six years, with GPT-5.5's TH reaching over 3 minutes and reasoning token horizon exceeding 1,500 tokens. Our median estimates predict that frontier no-CoT THs could exceed 7 minutes by 2028, and 25 minutes by 2030, though these projections carry substantial uncertainty. We recommend frontier developers track this explicitly.

1 Introduction

The paper asks how much frontier models can reason without externalized chain-of-thought and how that latent capacity changes over time. It evaluates no-CoT performance across diverse benchmarks and quantifies it using human time and reasoning-token horizons.

  • CoT monitoring can become harder if models perform complex reasoning internally rather than exposing it in intermediate output.
  • The study evaluates no-CoT reasoning across 43 benchmarks spanning short-answer, generative, and multi-turn agentic tasks.
  • The paper measures latent reasoning with 50%-task-completion time horizons and minimum anchor-model reasoning tokens.
  • Across 14 frontier models, no-CoT time horizons doubled every 373 days over six years, reaching over 3 minutes for GPT-5.5.
  • Median projections put no-CoT time horizons above 7 minutes by 2028 and 25 minutes by 2030, with substantial uncertainty.
  • No-CoT horizons doubled roughly two times more slowly than the with-CoT trend reported for full-affordance agentic tasks.

2 Related work

Related work covers CoT monitorability, latent-reasoning evaluations, and task-completion time horizons. This paper extends those lines by evaluating no-CoT capabilities across broad domains and comparing them with with-CoT trends.

  • CoT may expose information useful for monitoring undesirable reasoning, although monitoring can become harder with linguistic drift or encoded reasoning.
  • Prior latent-reasoning research includes continuous latent-space architectures, CoT distillation, and evaluations without thinking tokens.
  • This work evaluates LLMs prompted to complete tasks without CoT, alongside filler-token and problem-repeat experiments.
  • Task-completion horizons have been applied to long-horizon software and offensive-cybersecurity tasks, with later work examining domains and benchmark correlations.

3 Methods

The methods combine a broad no-CoT benchmark suite, calibrated human solve-time estimates, logistic task-completion horizons, and an anchor-model reasoning-token horizon. Hierarchical bootstrap resampling propagates uncertainty through model estimates and growth trends.

  • The evaluation covers 14 frontier models from GPT-2 to GPT-5.5 using benchmark-specific no-CoT elicitation and structured-output fallbacks.
  • The benchmark suite contains 43 benchmarks spanning math, coding, knowledge, reasoning, and safety-relevant tasks with human times from 0.5 seconds to over 15 hours.
  • Human solve times combine existing data, 733 expert problem attempts, and calibrated Claude Opus 4.7 estimates for benchmarks lacking measurements.
  • The 50% task-completion horizon is estimated by fitting a logistic success curve whose midpoint is the human solve time corresponding to 50% success.
  • Hierarchical bootstrap resampling propagates uncertainty across benchmarks, problems, model runs, and calibrated human solve times into horizon estimates and doubling trends.
  • A two-regime recalibration leaves the downstream doubling-time trend and R2 essentially unchanged.
  • The reasoning-token horizon uses the minimum successful token count from eight anchor-model attempts, with o3-mini selected as the anchor.

4 Results

Frontier models’ no-CoT reasoning capacity has grown exponentially across time and token horizons, with both doubling roughly annually. The trend is robust to several analysis choices, while absolute horizons vary across benchmarks and model architectures.

  • Main results: Doubling trends: 373 days: the 50% no-CoT time horizon doubling time, with a 95% CI of [167, 691].The 50% reasoning token horizon doubles every 437 days, with a 95% CI of [341, 571].
  • Main results: Doubling trends: Over 3 minutes: today’s frontier 50% no-CoT time horizon, equivalent to roughly 1.5k o3-mini reasoning tokens.GPT-5.5 has the longest estimated 50% no-CoT problem-completion horizon at 3 minutes.
  • Main results: Doubling trends: By 2028, the median projection reaches ∼7 minutes, and by 2030, ∼28 minutes, with wide projection ranges.The corresponding median token-horizon projections are ∼3.7k tokens in 2028 and ∼12k tokens in 2030.
  • Main results: Doubling trends: The no-CoT horizon doubling time is roughly twice as long as the with-CoT trend reported by Kwa et al.The divergence begins with the release of GPT-4.
  • Robustness of our results: Question repeats improve performance by 10–20 percentage points on certain benchmarks, whereas filler tokens have negligible impact for most model–benchmark combinations.Reported gains occur on N-hop, Sally-Anne, and Scheming (numeric), with effects varying across benchmarks.
  • Robustness of our results: Doubling times vary from 246 to 387 days across domains, but category estimates are noisy because several categories contain few benchmarks.The authors report that the overall exponential growth trend is robust across sensitivity analyses and alternative modelling assumptions.
  • Robustness of our results: Adding generation tasks leaves estimated horizons and doubling time broadly unchanged, while adding generation and multi-turn agentic tasks modestly increases horizons.The latter expansion reduces the estimated doubling time by around 50 days to 319 days.
  • Open-weight model parameter scaling: Doubling the 50% TH requires a 4.2× increase in total parameters, while dense and MoE models require 2.2× and 8.1× increases, respectively.Layer count is a relatively strong correlate of TH and has essentially the same slope across dense and MoE architectures.

5 Discussion and limitations

Frontier models can complete tasks requiring minutes of human effort without externalizing intermediate reasoning, and this no-CoT capacity has grown over time. The estimates are uncertain and may understate latent capabilities, while not directly measuring hidden deliberation.

  • No-CoT task-completion horizons have doubled roughly annually over the past six years.The direction is robust, but the 95% confidence intervals around doubling time are wide.
  • At the slowest doubling within the 95% CI, no-CoT horizons still reach almost 10 minutes by 2030.Median and upper-bound estimates are significantly higher, and reducing uncertainty remains a reason to continue measuring latent reasoning.
  • Current LLMs can solve tasks requiring minutes of human effort without externalizing intermediate reasoning.The results provide a lower-bound-style empirical estimate of reasoning that could occur without revealing it to a monitor.
  • The estimates depend on no-CoT elicitation and may systematically under-estimate latent capabilities when short or single-token answers are out-of-distribution.They also do not perfectly distinguish latent reasoning from memorisation or heuristic pattern-matching, so horizons do not directly measure hidden deliberation or planning depth.

6 Conclusion

The paper finds that frontier models can perform minute-scale tasks without externalizing reasoning, with no-CoT task-completion horizons doubling roughly annually since 2019. It recommends continued measurement, especially in safety-critical domains.

  • Frontier models can now perform tasks that take humans minutes without externalising their reasoning in CoT.
  • No-CoT task-completion horizons have doubled roughly every year since 2019.
  • Future work should continue measuring latent reasoning, especially in safety-critical and high-stakes domains.The paper urges frontier labs to track these trends because they can do so cheaply.

Contributions

The contribution materials identify the research and implementation work underlying the benchmark suite, experiments, and human solve-time calibration. Responsibilities span evaluation design, frontier experiments, reasoning-token anchors, and time baselines.

  • FRW led the research, wrote most of the paper, and implemented the hash function, Kenken, and Puzzle Baron evaluations.
  • DG co-led research, co-wrote most of the paper, ran full-scale frontier TH experiments, and implemented numerous benchmark evaluations.
  • ACW led reasoning-token anchor experiments and implemented ARC-AGI and Strategic Sche… evaluations.
  • IC led collection of human solve-time baselines and implemented the Intuit Physical evaluation.
  • Other contributors implemented the remaining benchmark evaluations, including arithmetic, coding, steganography, A-level, LinuxArena, GPQA Diamond, LingOly, GSM1k, TerminalBench, and Tower of London.

A.1 Time and Token Horizon Results

Across models from GPT-3 to GPT-5.5, no-CoT time and reasoning-token horizons grew by roughly two and three orders of magnitude, respectively. Horizon estimates use logistic success relationships with task length, though comparisons across benchmark suites require caution.

  • Per-model horizon results: GPT-3’s ∼0.05-second / 1-token baseline grew to GPT-5.5’s ∼3 minutes / 1.5k tokens from 2020 to 2026.This corresponds to roughly two orders of magnitude on the time axis and three on the token axis.
  • Per-model horizon results: GPT-5.5’s time horizon has a 95% bootstrap CI of [0.8, 62] min, while its token horizon has [950, 2.4k] tokens.Time intervals widen sharply with model strength, whereas token intervals are markedly tighter.
  • Success-rate figures: Success rates are grouped by human solve time and o3-mini reasoning-token count, with a very strong correlation between performance and reasoning token count.High-human-time artefacts for recent models are partly driven by tasks that are long for humans but well-suited to LLMs.
  • Comparison to full-affordance horizons: A 373-day no-CoT doubling time is twice as slow as METR’s 182-day doubling time for full-affordance agentic tasks.The comparison should be interpreted cautiously because the benchmark distributions differ.
  • Comparison to full-affordance horizons: No-CoT THs measure task completion without exposing intermediate reasoning, whereas full-affordance THs measure deployed agentic performance.The paper argues that both trends matter for understanding capability growth and reasoning monitorability.
  • Logistic slope analysis: The median logistic slope is 1.03 on the time axis and 0.64 on the token axis, indicating that human time is the stronger per-question difficulty discriminator.Slopes are interpreted through log-odds of failure per doubling of task length.

A.3 Robustness to the time-uncertainty noise model

The canonical trend is robust to replacing the single-Gaussian time-uncertainty model with a two-regime split-Gaussian model, but including GPT-2 and GPT-3 makes the doubling time depend on an arbitrary horizon floor.

  • Noise-model robustness: The split-Gaussian model uses separate calibration regimes below and above a 1-minute estimated solve time.The fitted residual scales are σlow = 1.06 for 311 observations and σhigh = 0.50 for 764 observations.
  • Horizon-floor sensitivity: GPT-2 and GPT-3 have horizons near or below methodological resolution, requiring an arbitrary floor when included in trend fitting.Their logistic-fit THs are approximately 2 ms and 2.85 s, respectively.
  • Noise-model robustness: The split-Gaussian and single-Gaussian analyses produce indistinguishable doubling times and confidence envelopes.This indicates that the canonical trend is insensitive to this particular choice of time-uncertainty model.
  • Horizon-floor sensitivity: The estimated doubling time changes monotonically with the selected floor when GPT-2 and GPT-3 are included.With both models excluded, the slope is essentially floor-invariant.

A.5 Trend line sensitivity analysis

Trend estimates are broadly robust to model and benchmark exclusions, while restricting the analysis to tasks with real human solve times lowers absolute horizons but increases a noisier doubling rate.

  • Model sensitivity: Removing any individual model leaves the doubling-time estimate within broadly compatible uncertainty, although excluding GPT-3.5 widens the error bars.The point estimate excluding GPT-3.5 increases substantially, but the broader interval contains the headline estimate.
  • Human-time sensitivity: Using only tasks with real human solve times slightly reduces 50% TH values but increases the doubling time.The restricted dataset contains fewer benchmarks, making these estimates very noisy.
  • Benchmark sensitivity: Leaving out any single benchmark changes the TH doubling time by at most an amount within the headline error bars.Some exclusions increase the estimate slightly, while others decrease it.

A.8 Effect of longer tasks

Adding longer-form generation tasks leaves short-answer trend estimates largely unchanged, whereas including multi-turn agentic tasks substantially raises frontier-model TH estimates; category doubling times remain noisy and varied.

  • Generation tasks: Adding longer-form generation tasks produces statistically insignificant increases in TH and reasoning-token doubling times, with essentially unchanged per-model estimates.The headline analysis otherwise restricts the main results to short-answer benchmarks.
  • Agentic tasks: Including multi-turn agentic tasks most noticeably increases GPT-5.5’s TH estimate, which nearly doubles because of strong performance on very long-horizon agentic tasks.These tasks are excluded from the main results because they differ substantially from short-answer single-turn tasks.
  • Task categories: Category-specific TH doubling times range from 246 days for math and science to 387 days for language and reasoning.These estimates are highly noisy because each category contains fewer tasks and benchmarks.
  • Trend specification: An exponential fit is preferred over hyperbolic and linear alternatives because it fits better, is more stable under bootstrap refits, and has a direct doubling-time parameterization.The reported fit qualities are R2 = 0.87 for exponential, 0.82 for hyperbolic, and 0.46 for linear.

A.11 Time and reasoning token horizon per benchmark

Per-benchmark horizon estimates are obtained with filters for interpretability and generally cluster around each model’s pooled horizon, supporting a single-horizon summary while allowing benchmark-level variation.

  • Estimation procedure: The per-benchmark procedure repeats each model’s pooled fitting method separately for every benchmark to estimate both 50% TH and reasoning token horizons.Filters exclude unstable fits caused by small benchmark size, near-saturation, or insufficient score variation.
  • Trend comparison: The paper compares exponential, hyperbolic, and linear fits to median per-model TH estimates.The comparison concerns the trend specification rather than the per-benchmark fitting procedure.
  • Cross-benchmark consistency: Per-benchmark estimates generally cluster within roughly one order of magnitude of each model’s pooled horizon.This consistency supports treating a single horizon as a meaningful model-level summary statistic.
  • Visualization: The figures organize estimates with one panel per model and one row per benchmark, coloring bars by mean human task time or mean reasoning tokens.The two visualizations separately show TH and reasoning token horizon estimates.

A.12 Robustness to minimum-reasoning token floor

The token-anchor trend is robust to excluding questions with zero minimum reasoning tokens, while filler tokens and question repeats produce asymmetric effects across aggregate and benchmark-level results.

  • Robustness to token floor: 454 days versus 437 days: dropping approximately 6% of zero-token questions leaves the token-anchor doubling time statistically indistinguishable from the canonical estimate.The refit’s R2 is unchanged, indicating insensitivity to this floor choice.
  • TH and token horizons: R2 = 0.95 with slope 0.85: TH and reasoning token horizons follow a strong log-log relationship across frontier models.A 10× token-horizon increase corresponds to roughly a 7× increase in TH.
  • Fillers and repeats: Filler tokens have practically no aggregate effect, whereas one additional question copy yields a small improvement that saturates through 50 repeats.The filler sweep uses deterministic counting sequences, while repeats preserve the original question text and vary only its number of appearances.
  • Benchmark-level variation: Benchmark-level effects diverge: filler tokens significantly harm Monitor Poisoning, while repeats harm Monitor Poisoning but consistently improve N-hop lookup, Sally-Anne, and Scheming (Numeric).For Sally-Anne, GPT-5.4 gains with at least 50 fillers, while Opus models decline.

A.17.5 TH vs. Artificial Analysis intelligence score

The paper compares 50% task-completion time horizons with Artificial Analysis Intelligence Index scores using the non-reasoning score column. A doubling in TH corresponds to a 1.7× increase in AA Index score.

  • TH versus AA Index: A doubling in the 50% TH corresponds to a 1.7× increase in AA Index score.The analysis uses AA Index v4.0 and the non-reasoning column for consistency across models.

A.17.6 TH in dense and MoE models

Dense and MoE models show different no-CoT TH scaling with parameter count, but similar scaling with layer count. Several comparisons are sensitive to sparse data, outliers, and model-selection effects.

  • Total parameters: 2.2× versus 8.1×: doubling TH requires far more total parameters for MoE models than dense models.MoE models activate only 3–20% of their parameters per forward pass.
  • Active parameters: 13× versus 2.2×: doubling TH requires 13× more active parameters for MoEs, compared with 2.2× for dense models.The authors note that the MoE Pareto frontier contains only three models, making this estimate fairly unreliable.
  • Layer count: 1.29× versus 1.27×: dense and MoE models have nearly identical TH doubling factors when scaling is plotted against layer count.Dense models nevertheless have approximately 1.7× higher TH at a given layer count.
  • Reasoning-model comparisons: 6.6×: removing Gemma 4 31B reduces the hybrid-reasoning doubling factor and raises R2 from 0.68 to 0.93.This indicates that the hybrid-reasoning trend is strongly influenced by that outlier.
  • Other comparisons: No significant TH-scaling difference appears between text-only and vision-language models, while Qwen 3, Gemma 3, and Ministral show clean within-family scaling.Their within-family R2 values range from 0.97 to 1.0.
  • Population-level predictors: Total parameter count and layer count are the strongest predictors of no-CoT TH, while training FLOPs are the weakest among the studied axes.The qualitative ranking is more robust than the absolute correlation magnitudes.

B.3.2 Per-benchmark calibration

In-domain examples substantially improve model-based human-time estimates, whereas cross-domain examples provide little benefit. The calibration analysis also documents constraints on benchmark coverage and reasoning-token interpretation.

  • In-domain calibration: MALR falls from 1.47 zero-shot to 0.59 with 10 in-domain examples, a ∼60% reduction.Performance continues improving where data are available, reaching approximately 0.54 by N=100.
  • Cross-domain calibration: 10 cross-domain examples reach MALR 1.36, only an 8% improvement over the 1.47 zero-shot baseline.The curve plateaus by N=3.
  • Per-benchmark results: In-domain few-shot outperforms zero-shot and cross-domain methods across all 20 benchmarks except the two SHADE-Monitor splits.Those splits allow only N ≤3 in-domain examples, making results statistically indistinguishable from zero-shot.
  • Comparison with heuristics: Mean MALR is 0.47 for in-domain few-shot versus 0.71 for ad-hoc heuristics across 14 benchmarks with both estimates.In-domain few-shot beats the heuristic on 11 of 14 benchmarks.
  • Pooled calibration: Pooled MALR decreases from 1.23 for zero-shot estimates to 0.50 for best few-shot estimates.Best few-shot predictions tighten scatter and better capture extreme outliers.
  • Reasoning-token anchor: Reasoning-token counts are model-dependent and variable, so they serve as a relative task-difficulty reference rather than a direct measure of hidden computation.The anchor uses minimum reasoning-token counts among successful attempts and selects o3-mini for its strongest negative correlation with no-CoT success and low inference cost.
  • Uncertainty: The minimum-token estimator has sampling noise because only k attempts may be available for each question.The analysis bootstraps correct attempts and reports relative standard deviations of the resulting minima.

C.3 Data collection app

The study used a web app to collect participant answers and measure solve times for benchmark problems.

  • Data collection: The web app recorded participant sign-up, problem selection, answer submissions, and solve times.The interface also tracked active solve time, attempt numbers, and grading status.

D The benchmark suite

The benchmark suite spans diverse domains and task difficulties, combining existing and newly constructed datasets with standardized final-answer evaluation and human-time estimates.

  • 43 benchmarks cover recall, reasoning, coding, and safety-relevant domains across a wide range of task difficulties.
  • The suite includes real UK A-Level questions converted into independently scored, correctness-only tasks with preserved visual materials and provenance.Prompts remove generic exam instructions, inline dependencies, and retain task-specific visual assets.
  • Human timing coverage combines official paper durations and mark-based proxies for A-Level tasks with recruited-participant solve times for other benchmarks.For A-Level tasks, marks directly define lower, median, and upper time estimates; ARC-style benchmarks use recruited human baselines and percentile time ranges.
  • Evaluation generally uses binary final-answer grading, with benchmark-specific scorers such as exact grid matching for abstract-reasoning tasks.The A-Level benchmark reports task accuracy and marks-weighted percentage, while grid tasks receive no partial credit.
  • The suite also includes abstract grid-transformation tasks designed to be difficult for AI but comparatively easy for humans.Questions provide input-output examples and require predicting the output grid for a held-out input.
  • LingOly-TOO applies linguistic obfuscation to reduce memorization benefits while preserving the underlying translation logic.The obfuscations permute target-language forms so that languages appear new to models.
Loading 2606.07157v4…