PaperlayerView exact revision on arXiv

Source-linked AI summary

From BERT to Frontier Agents: Eight Years of Language-Model Progress, the Collapse of the Capability-Cost Curve, and the Rise of Task-Targeted Models

Pranav Kumar Kaliaperumal

arXiv:2608.13675v1cs.LGcs.AI

TL;DR

The paper asks how model capability and cost efficiency changed, and how much careful inference-time compute can extract from fixed model weights. It analyzes benchmark and pricing series and reports rapid coding gains alongside task-specific model leaders.

  • Problem

    The paper asks how frontier capability and cost efficiency changed and how much inference-time compute can improve performance from fixed model weights.

    Source: §1 · p. 3

  • Method

    The paper compares public benchmark and pricing series and assesses fixed-weight gains from inference compute, confidence estimation, and routing across models.

    Source: §1 · p. 3

  • Results

    Different models lead across tasks, while SWE-bench Verified task-solve odds show ∼5.8× annual growth.

    Source: §1 · p. 3, §7 · p. 13

  • Takeaways & Limitations

    For buyers, procurement increasingly calls for a routing policy with per-task effort settings rather than a single model choice.

    Source: §6 · p. 13

    Task-targeted gains on a benchmark can overstate transfer to new settings.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

Between October 2018 and July 2026 AI models progressed from simple systems like BERT to massive agents that solve complex math and write software. The ability to resolve real coding issues improved by nearly six times per year since late 2024. During this time costs dropped sharply with OpenAIs budget model GPT 5 point 6 Luna matching flagship capabilities for just one to six dollars per million tokens beating older versions at a fraction of the price. Top performance is now split across specialized models as Claude Opus 5 leads in frontend coding Claude Fable 5 excels at repository level coding and GPT 5 point 6 Sol dominates terminal tasks. In a grade school math test using the Qwen 2 point 5 model basic methods solved 58 of 100 problems while advanced sampling solved up to 79. A confidence ranking tool correctly identified 47 right answers in its top 50 choices proving highly useful for sorting tasks with all research materials made fully public.

1 Introduction

The paper traces language models from BERT’s 2018 reading-model era to 2026 reasoning models and tool-using agents, asking how capability and cost changed and what careful inference and model routing can add.

1 Introduction

  • The paper provides a sourced 2018–2026 timeline across three benchmark eras, tracing a shift from task-specific fine-tuning to reasoning models and tool-using agents.BERT set the state of the art on eleven natural-language-understanding tasks, while Claude Opus 5 reported 42/42 on the 2026 IMO problems under a post-contest, no-tools protocol.Source: §1 · p. 3
  • ∼5.8× annual growth in the odds of solving SWE-bench Verified tasks coincided with saturation of knowledge benchmarks such as MMLU.The paper reports trend fits for coding-task odds and separately notes saturation in knowledge benchmarks.Source: §1 · p. 3
  • ∼60× decline in input-token prices from GPT-3 (2020) to GPT-5.6 Luna (2026) accompanied a budget tier matching the prior quarter’s flagship on most agentic/professional benchmarks.The cost analysis compares public pricing and benchmark series.Source: §1 · p. 3
  • The 2026 frontier is task-fragmented, and simple model routing recovers the per-task optimum.The paper examines routing work across models as a way to obtain task-specific performance.Source: §1 · p. 3
  • A locked development/evaluation study tests self-consistency for a small open model, followed by an exploratory confidence model that ranks predictions for triage.The accompanying analyses include complete traces, paired uncertainty, and reasoning effort.Source: §1 · p. 3

2 Eight years in four paradigms

Language-model progress moved from fine-tuning task-specific heads to eliciting capabilities from frozen models, then to aligned instruction following and reasoning agents. By 2026, real coding issues and other long-horizon tasks had become central measures of capability.

2.1 The encoder era: BERT and the fine-tuning paradigm (2018–2019)

  • BERT1 established pretraining followed by a task-specific head; RoBERTa4 reached 88.5 on GLUE with more data and training, while T55 reached 90.3 on SuperGLUE with text-to-text training.GPT-26 showed that a sufficiently trained left-to-right decoder could acquire tasks without fine-tuning, though its zero-shot quality was not yet competitive.Source: §2.1 · p. 3

2.2 Scale and in-context learning: GPT-3 (2020–2021)

  • GPT-37 scaled decoders to 175B parameters and showed few-shot prompting could substitute for gradient updates, but its five-shot MMLU score was 43.9% and its 71.8 SuperGLUE score trailed fine-tuned systems [8].Its unaligned samples were unreliable, but it established that capabilities could be elicited from a frozen general model.Source: §2.2 · p. 4

2.3 Alignment and instruction following (2022–2023)

  • InstructGPT9 used reinforcement learning from human feedback for instruction following; GPT-4 reached 86.4% on MMLU [10], while Llama opened weights and Claude and Gemini established a multi-vendor frontier.ChatGPT packaged instruction-following models for general users, and capability reports became a modern release pattern.Source: §2.3 · p. 4

2.4 Reasoning and agents (2024–2026)

  • SWE-bench Verified scores on independent harnesses rose from 49% for Claude 3.5 Sonnet (October 2024) to 97% for Claude Opus 5 (July 2026) [12, 13, 14].The benchmark made resolving real GitHub issues a standard hard problem.Source: §2.4 · p. 4
  • OpenAI’s o1 showed that longer trained reasoning traces and additional inference tokens could lift math and coding performance, scoring 91.8% on MMLU [11].Its score effectively saturated that benchmark.Source: §2.4 · p. 4
  • By mid-2026, flagship releases were evaluated primarily on agentic, long-horizon, and economically weighted tasks; Kimi K3’s 2.8-trillion-parameter open-weight model was competitive with leading proprietary systems [15–18].Kimi K3 marked the near-closure of the open–closed gap [17].Source: §2.4 · p. 4

3 Data and methods

The study combines public model and pricing evidence with a locked GSM8K evaluation and reproducible traces, then applies trend, cost, routing, and exploratory confidence analyses. Its methods distinguish vendor-reported claims from independent measurements.

3.1 Data

  • The data include 27 timeline rows, a July 2026 frontier score-and-price matrix, representative 2020–2026 API prices, and a locked 100-item GSM8K evaluation split.The repository also contains three analysis tables and complete traces for reproducing reported results.Source: §3.1 · pp. 4–5
  • The reproducibility architecture connects public evidence to deterministic trend, cost, and routing analyses, and uses versioned live-evaluation traces for the exploratory confidence stage and auditable outputs.The live evaluation records complete traces and paired statistics.Source: Figure 1 · p. 5
  • Mid-2026 figures mix vendor-reported and independent evaluations; the study labels provenance, prefers third-party measurements when available, and treats launch figures as vendor claims.It notes where vendor and third-party evaluation harnesses differ.Source: §3.1 · p. 5

3.2 Analysis methods

  • The analyses fit log-odds trends for SWE-bench Verified, compare MMLU with an estimated 89.8% human-expert level, compute the GPT-5.6 family cost–capability Pareto frontier, and compare single-model, oracle, and restricted routing.The routing comparison uses 14 benchmarks reported for six models; the Pareto analysis uses 15 model-by-effort settings.Source: §3.2 · p. 6
  • The frozen Qwen2.5-1.5B-Instruct configuration used four-sample plurality at temperature 0.4 on 100 disjoint GSM8K evaluation items, with paired greedy comparisons and exact-count and interval reporting.The protocol also recorded sampled completions, answers, votes, verifier decisions, and correctness, and specified exact McNemar tests against paired greedy outcomes.Source: §3.2 · p. 6
  • The exploratory confidence model uses seven reference-free features and five-fold stratified cross-validation, but was designed after v1 outcomes and is not a second held-out evaluation.It reports ROC AUC, Brier score against a 0.62 prevalence baseline, and accuracy at fixed 50% and 75% coverage.Source: §3.2 · p. 6

4 Results

Benchmarks show rapid capability gains and falling API prices, but performance now varies by task: no model leads everywhere, and benchmark scores may overstate transfer to new settings.

4.1 Capability growth: three benchmark eras

  • MMLU rose from 43.9% (GPT-3, 2020) to 91.8% (o1, late 2024), past the 89.8% estimated human-expert level, then lost discriminative power.Four strong models clustered within 4 points; the field responded with harder knowledge tests and increasingly agentic tasks. GLUE remained discriminative for about two years and MMLU for roughly four.Source: §4.1 · pp. 6–7
  • SWE-bench Verified solve odds grew ~5.8× per year (R2 = 0.78, n = 14), while independent evaluation puts the frontier at 96–97%, prompting harder variants.The best model scores only 43–64% on SWE-bench Pro and Frontier-Bench. SWE-bench Verified appears headed toward saturation within roughly a year.Source: §4.1 · p. 7, Figure 2 · p. 7

4.2 The collapse of the capability–cost curve

  • API input prices fell ~60×, from $60 per million tokens (GPT-3, 2020) to $1 (GPT-5.6 Luna, 2026), while capability compounded at a similar rate.The effects multiply: a dollar buys orders of magnitude more capability than a dollar in 2023.Source: §4.2 · p. 7
  • GPT-5.6 Luna, priced at $1/$6 per million input/output tokens, won 9 and lost 14 of 23 shared benchmarks against GPT-5.5 at $5/$30.Luna won 7 of 10 agentic and professional-work benchmarks; its reconstructed SWE-bench Verified score was 92.9% versus GPT-5’s 74.9%. It cost ~20% of the flagship token price but gave up long-context recall (41.3% vs. 81.5% on MRCR) and the hardest academic reasoning.Source: §4.2 · pp. 7–8
  • Every Terra configuration was dominated in the GPT-5.6 family Pareto analysis, leaving a bimodal structure of volume and premium tiers.Six of fifteen settings lay off the frontier; a Luna or Sol setting was always smarter at no extra cost or equally smart and cheaper.Source: §4.2 · p. 8

4.3 The frontier fragmented by task

  • Task leadership split: Opus 5 led frontend coding, Fable 5 repository-level coding, and GPT-5.6 Sol agentic terminal and long-horizon professional work.Opus 5 led WebDev at 1,712 Elo, while Kimi K3 was second at 1,682; both values were preliminary. Fable 5 scored 80.0% on SWE-bench Pro versus Sol’s 64.6%; Sol led the Coding Agent Index (80), Terminal-Bench 2.1 (88.8%; 91.9% in multi-agent Ultra mode), and Agents’ Last Exam (52.7%, +12.2 over the best non-OpenAI model).Source: §4.3 · p. 8
  • Opus 5 also led novel reasoning and computer use, while reaching within 0.5% of Fable 5’s CursorBench peak at half the task cost.Opus 5 scored 30.2% on ARC-AGI-3 versus the previous record of 7.8% (GPT-5.6 Sol), reported 42/42 on IMO 2026, and led on OSWorld 2.0.Source: §4.3 · p. 8
  • On fourteen fully covered benchmarks, Sol averaged 97.6% of the per-benchmark best, while an oracle router averaged 100% (+2.4 points).A two-model Sol–Fable router captured the entire oracle gain; a cost-constrained router matched the oracle score at 95% of its cost.Source: §4.3 · p. 8

5 Improving fixed models at inference time

The section evaluates sampling, confidence-based selection, workload-matched reasoning effort, and routing among specialized models. Sampling raised observed accuracy modestly but did not establish a population effect, while confidence triage, effort tuning, routing, and reporting recommendations address different deployment tradeoffs.

5.1 Sample, then aggregate

  • 58/100 (58.0%; Wilson 95% CI [48.2, 67.2]) greedy versus 62/100 (62.0%; Wilson 95% CI [52.2, 70.9]) four-sample plurality; paired exact McNemar p = 0.481.The observed four-point gain was not an established population effect: confidence intervals overlapped and the paired test did not reject equality. At least one sample was correct on 79/100 items, leaving a 17-point gap between the 62/100 plurality result and the oracle ceiling.Source: §5.1 · p. 10, Table 3 · p. 11

5.2 Estimate confidence for selective use

  • 47/50 (94.0%) of the top-ranked vote-at-four predictions were correct under out-of-fold confidence scoring.The top 75 contained 53 correct answers (70.7%), while full coverage returned to 62/100. Because the same 100 labels trained and assessed the cross-validated model, independent data must confirm the result.Source: Table 4 · p. 11, §5.2 · p. 11

5.3 Match reasoning effort to the workload

  • 44.4% was Opus 5’s Frontier-Bench peak at xhigh effort, versus 43.3% at max; each roughly four-point model gain multiplied per-task cost by 2.6× and then 1.9×.The source also reports a community re-analysis placing the Frontier-code peak near medium effort at ∼53%. It recommends measuring effort per workload rather than treating maximum effort as a universal quality setting.Source: §5.3 · pp. 11–12

5.4 Route work across specialized models

  • 97.6% of the per-benchmark maximum was the average for the best individual model across fourteen fully covered benchmarks; a Sol–Fable 5 combination captured the oracle’s full gain.The passage recommends complementary frontier models for quality-sensitive work and a budget tier for well-defined subtasks at lower cost.Source: §5.4 · p. 12

5.5 Reporting priorities for model builders

  • Model builders should publish per-benchmark score–cost curves, pair targeted benchmarks with held-out transfer suites, and prioritize mid-context retrieval over longer context windows alone.Luna scored 41.3% on MRCR against Sol’s 91.5%, its clearest break from its near-flagship pattern; the ARC-AGI-3/Witness gap also motivates held-out transfer suites.Source: §5.5 · p. 12

6 Discussion: how models get good at specific tasks

The discussion attributes task-specific model strengths to targeted training, architecture, and inference-time structure, while warning that benchmark gains may not transfer and calling for more rigorous evaluation.

Domain-weighted post-training

  • Domain-weighted reinforcement learning on verifiable, task-specific rewards lets labs buy capability in selected verticals, including code, math, and agent tasks.Moonshot’s agentic suite results and OpenAI’s terminal/agentic lead track heavy reinforcement-learning investment in those task families16,17.Source: §6 · p. 12

Architecture as targeting

  • Kimi K3’s hybrid linear attention and expert routing target long-context agentic efficiency, with BrowseComp 91.2 reported as state of the art at release17.Source: §6 · p. 12

Inference-time structure

  • GPT-5.6 Ultra’s multi-agent fan-out raises Terminal-Bench from 88.8% to 91.9% at higher cost16, while confidence-based routing can direct extra compute or review to uncertain answers.The paper relates its small-scale self-consistency experiment to the same inference-time lever.Source: §6 · p. 12

Benchmark-shaped optimization and its limits

  • A ∼4× ARC-AGI-3 record for Opus 5 did not yield a win on held-out puzzles, while SWE-bench passes increasingly diverged from maintainer acceptance [26,27,31].The held-out puzzles included an environment where Opus 5 stated hidden rules before its first move; the passage says this suggests familiarity, not just reasoning. Benchmarks can also saturate, as GLUE and MMLU have, with SWE-bench Verified nearing saturation.Source: §6 · p. 13

Implication for evaluation

  • The paper calls score-versus-cost curves, held-out transfer suites, and maintainer-grade review minimum standards for evaluating model claims.It notes that vendors report curves18, ARC Prize independently administers runs with published traces3,27, and METR’s maintainer study offers a template for auditing benchmark passes.Source: §6 · p. 13

Implication for buyers

  • A budget tier at 20% of flagship token price matches last quarter’s flagship on most professional work, making task-specific routing—not a single model—the procurement unit.The passage frames the buyer’s question as which combination of models and effort settings to use for each task.Source: §6 · p. 13

Limitations

  • The reported GPT-5.5 score is 85.1% in the vendor report versus 82.6% in an independent-harness reconstruction, and the 5.8× SWE-bench trend spans only 22 months14,25.The trend mixes vendor and independent evaluations; provenance labels cannot fully remove harness differences, so 5.8× indicates scale rather than a stable law. The routing oracle knows benchmark identity, unlike a production router facing unlabeled traffic.Source: §6 · p. 13
  • The v1 confidence intervention used 16 development items and one 100-item GSM8K evaluation; its four-point primary gain was not statistically significant (p = 0.481).It covers one model family, task, and sampling seed, with broad Wilson intervals. Designed after labels were visible and cross-validated on the same 100 items, its AUC and 94% top-half accuracy remain exploratory; seeded candidates differ between CPU and CUDA, and public benchmark figures are current as of August 10, 2026.Source: §6 · p. 13

7 Conclusion

Across eight years, language-model progress shifted through successive capability regimes, while agentic coding improved and the cost of fixed capability fell sharply. Fixed-model sampling and confidence-based selection also show potential for GSM8K gains, and the paper releases materials for reproducibility.

7 Conclusion

  • Eight years separate BERT’s 80.5 GLUE score from Opus 5’s reported 42/42 on the 2026 IMO problems.Progress moved through fine-tuning, few-shot scale, alignment, reasoning, and agency rather than one smooth exponential, with each shift exhausting the previous era’s benchmarks.Source: §7 · p. 13
  • Agentic-coding solve odds are rising roughly six-fold per year, while a $1/M budget tier reproduces most professionally relevant performance of an elevenweek-old flagship.Frontend coding, repository repair, terminal work, and novel reasoning have different model leaders, making deployment increasingly dependent on routing policy rather than one model choice.Source: §7 · p. 13
  • Four-sample plurality moved frozen GSM8K exact match from 58% to 62% (p = 0.481), while at least one sample was correct on 79% of items.An exploratory logistic model using answer agreement and trace statistics reached out-of-fold AUC 0.833, with 47 of its 50 highest-confidence predictions correct; this supports selective triage, not full-coverage accuracy above 62%, and needs confirmation on untouched data.Source: §7 · p. 14
  • The accompanying archive includes datasets, analysis scripts, locked manifests, inference traces, unit tests, figure generation, and the paper build script.Reproduction requires no proprietary access beyond live benchmark numbers, whose sources and retrieval dates are in code/data sources.md.Source: §7 · p. 14

Related papers

Loading 2608.13675v1…