PaperlayerView exact revision on arXiv

Source-linked AI summary

From BERT to Frontier Agents: Eight Years of Language-Model Progress, the Collapse of the Capability-Cost Curve, and the Rise of Task-Targeted Models

Pranav Kumar Kaliaperumal

arXiv:2608.13675v1cs.LGcs.AI

TL;DR

The paper addresses how capability, cost efficiency, and inference strategies changed from BERT to frontier agents. It finds roughly six-fold annual growth in agentic-coding solve odds alongside a fragmented frontier increasingly requiring task-specific routing.

  • Problem

    The paper examines how capability and cost efficiency evolved from BERT to frontier agents and how inference compute, confidence, and routing can improve fixed models.

  • Method

    It analyzes public benchmark and pricing series, then evaluates inference-time sampling, confidence estimation, and routing across models using reproducible released materials.

  • Results

    Agentic-coding solve odds rose roughly six-fold per year, while different models led frontend coding, repository repair, terminal work, and novel reasoning.

  • Takeaways & Limitations

    Deployment increasingly depends on routing policy and per-task effort settings rather than selecting a single universally best model.

  • Takeaways & Limitations

    Benchmark-targeted gains can overstate transfer, as Opus 5’s ARC-AGI-3 improvement did not clearly transfer to held-out Witness puzzles.

Abstract

from arXiv · show

Between October 2018 and July 2026 AI models progressed from simple systems like BERT to massive agents that solve complex math and write software. The ability to resolve real coding issues improved by nearly six times per year since late 2024. During this time costs dropped sharply with OpenAIs budget model GPT 5 point 6 Luna matching flagship capabilities for just one to six dollars per million tokens beating older versions at a fraction of the price. Top performance is now split across specialized models as Claude Opus 5 leads in frontend coding Claude Fable 5 excels at repository level coding and GPT 5 point 6 Sol dominates terminal tasks. In a grade school math test using the Qwen 2 point 5 model basic methods solved 58 of 100 problems while advanced sampling solved up to 79. A confidence ranking tool correctly identified 47 right answers in its top 50 choices proving highly useful for sorting tasks with all research materials made fully public.

1 Introduction … 4 Results

The paper is organized across four sections covering eight years of language-model paradigms, followed by data, methods, and results on capability growth, costs, and task specialization.

  • 1 Introduction: The introduction frames four periods: BERT’s encoder era, GPT-3’s scaling and in-context learning, alignment and instruction following, and reasoning and agents.These subsections span 2018–2026.
  • 2 Eight years in four paradigms: The paper’s second section examines eight years in four paradigms.The supplied outline identifies this section but does not provide further substantive findings.
  • 3 Data and methods: The data subsection appears before the analysis-methods subsection in the paper’s methodology structure.The outline places Data on page 4 and Analysis methods on page 6.
  • 4 Results: The results section reports capability growth across three benchmark eras.This subsection begins on page 6.
  • 4 Results: The results also examine the collapse of the capability–cost curve.This subsection begins on page 7.
  • 4 Results: Finally, the results analyze how the frontier fragmented by task.This subsection begins on page 8.

5 Improving fixed models at inference time

Section 5 presents five inference-time strategies for improving fixed models: sampling and aggregation, confidence estimation, workload-matched reasoning effort, routing across specialized models, and reporting priorities for model builders.

  • 5 Improving fixed models at inference time: Sample and aggregate model outputs at inference time.
  • 5 Improving fixed models at inference time: Estimate confidence to support selective use of model outputs.
  • 5 Improving fixed models at inference time: Match reasoning effort to the workload.
  • 5 Improving fixed models at inference time: Route work across specialized models.
  • 5 Improving fixed models at inference time: Define reporting priorities for model builders.

6 Discussion: how models get good at specific tasks · 7 Conclusion · 1 Introduction

The paper traces language-model progress from BERT to the 2026 frontier, measuring capability, cost efficiency, task specialization, and inference-time improvements using public benchmarks, pricing series, and released analyses. It reports rapid software-engineering gains, sharply lower token prices, task-fragmented performance, and benefits from self-consistency, confidence ranking, and routing.

  • 1 Introduction: 2018 BERT reached 80.5% on GLUE and state of the art on eleven language-understanding tasks, but could not write a coherent paragraph.BERT was a reading model requiring task-specific fine-tuning.
  • 1 Introduction: The study asks how models reached the 2026 frontier, how capability and cost efficiency changed, and how inference-time methods improve fixed model weights.The methods include calibrated reasoning effort, confidence estimation, and routing work across models.
  • 1 Introduction: The paper consolidates a sourced 2018–2026 capability timeline across three benchmark eras.This timeline is presented in Section 2 and Table 1.
  • 1 Introduction: ∼5.8× annual growth occurred in the odds of solving SWE-bench Verified tasks, while knowledge benchmarks such as MMLU approached saturation.The trend fits cover the period analyzed in the paper’s benchmark series.
  • 1 Introduction: ∼60× decline in input-token prices occurred from GPT-3 in 2020 to GPT-5.6 Luna in 2026.The cost analysis also reports that the budget tier matches the flagship of one quarter earlier on most agentic and professional benchmarks.
  • 1 Introduction: 2026 frontier performance is task-fragmented, and simple model routing recovers the per-task optimum.The paper therefore evaluates model selection by task rather than assuming one model dominates universally.
  • 1 Introduction: Self-consistency was studied in a locked development/evaluation setting for a small open model, followed by exploratory confidence-based prediction ranking for triage.The accompanying materials include complete traces, paired uncertainty, and reasoning-effort analyses.

2 Eight years in four paradigms

Language-model progress moved from fine-tuned encoders and task-specific heads to prompting, instruction-following, and test-time reasoning. By July 2026, agentic coding performance had reached 97% on SWE-bench Verified, up from 49% in October 2024.

  • Pretraining and fine-tuning: 88.5 on GLUE and 90.3 on SuperGLUE marked the encoder era, progressing from BERT’s pretrain-then-fine-tune recipe to RoBERTa and T5.BERT used masked-language-model pretraining followed by a task-specific head; T5 unified tasks as text-to-text.
  • Few-shot prompting: 43.9% five-shot MMLU showed that GPT-3 could elicit capabilities from a frozen general model, though it remained below fine-tuned systems on SuperGLUE.GPT-3 scaled the decoder recipe to 175B parameters and used a handful of in-context examples instead of gradient updates.
  • Instruction following and frontier models: 86.4% on MMLU brought GPT-4 near estimated human-expert level, while RLHF, ChatGPT, open weights, and multiple vendors established the modern ecosystem.InstructGPT introduced RLHF for instruction following; Meta’s Llama series opened weights, while Claude and Gemini created a multi-vendor frontier.
  • Reasoning and agency: 97% on SWE-bench Verified for Claude Opus 5 in July 2026 followed 49% for Claude 3.5 Sonnet in October 2024, as test-time compute and agency reshaped hard tasks.OpenAI’s o1 reached 91.8% on MMLU through longer trained reasoning traces and additional inference tokens; SWE-bench Verified framed the challenge as resolving real GitHub issues.

3 Data and methods

The study combines versioned public datasets, locked live-evaluation splits, and complete computational traces with explicit provenance controls. It applies trend, cost–capability, routing, live-evaluation, and post hoc confidence analyses.

  • Data and reproducibility: The repository contains three analysis tables, a locked evaluation split, and complete traces reproducing every reported result.The reproducibility architecture routes public evidence and live traces into auditable paper outputs.
  • Data construction: Twenty-seven timeline rows span 2018–2026, while the frontier matrix records benchmark scores and API prices for July 2026 models.Evaluation variants, including vendor-reported and independent SWE-bench Verified results, are recorded explicitly.
  • Live-evaluation data: 1,319 GSM8K test pairs anchor the live evaluation, with 16 pilot identifiers locked for development and 100 identifiers sampled using random state=20260810.The split manifest records both identifier lists and the source-parquet SHA-256 digest.
  • Provenance: Mid-2026 results mix vendor-reported and independent evaluations, so the study labels provenance and prefers third-party measurements when both exist.Vendor launch numbers are treated as claims rather than independent results, with differences between vendor and third-party harnesses noted.
  • Analysis methods: The analyses fit SWE-bench trend growth, compute GPT-5.6 cost–capability Pareto frontiers, compare routing strategies across 14 benchmarks, and evaluate Qwen2.5 prompting and confidence.The confidence model uses seven standardized reference-free features in L2-regularized logistic regression after frozen evaluation labels were available.

4 Results

Results show rapid progress in agentic coding alongside benchmark saturation and a sharp collapse in capability costs. Frontier leadership is task-dependent, making model routing valuable while task-targeted gains may not transfer reliably.

  • Benchmark progress: MMLU rose from 43.9% for GPT-3 to 91.8% for o1, exceeding the 89.8% estimated human-expert level before losing discriminative power.Four strong models clustered within 4 points by 2024.
  • Benchmark progress: 5.8× annual growth in SWE-bench solve odds drove agentic coding to the frontier, with current performance at 96–97% and saturation expected within roughly a year.The fit reports R2 = 0.78 across n = 14 observations.
  • Capability-cost curve: $60 to $1 per million input tokens marked a ∼60× API-price decline from GPT-3 to GPT-5.6 Luna, while capability compounded at a similar rate.A dollar therefore buys orders of magnitude more capability than in 2023.
  • Capability-cost curve: GPT-5.6 Luna costs $1/$6 per million input/output tokens versus GPT-5.5’s $5/$30 and wins 7 of 10 launch-table evaluations.Luna scored 92.9% versus GPT-5’s 74.9% on reconstructed SWE-bench Verified, but trails on long-context recall and hardest academic reasoning.
  • Task-targeted models: Claude Opus 5 leads frontend coding, Claude Fable 5 leads repository coding at 80.0%, and GPT-5.6 Sol leads terminal work at 88.8% on Terminal-Bench 2.1.No single model leads across the 2026 frontier; Sol reaches 91.9% in multi-agent Ultra mode.
  • Task-targeted models: Opus 5’s 30.2% ARC-AGI-3 score nearly quadruples GPT-5.6 Sol’s 7.8%, but its 43.4 Witness result is statistically tied with K3 and Fable 5.Benchmark-targeted gains can overstate transfer to new settings.
  • Task-targeted models: An oracle router averages 100% of per-benchmark best performance versus 97.6% for GPT-5.6 Sol, while a two-model Sol–Fable router captures the entire oracle gain.Task specialization improves routing because the models’ strengths are complementary.

5 Improving fixed models at inference time

Inference-time improvements raise fixed-model performance through repeated sampling, workload-matched effort, and model routing, but selection quality and empirical validation remain limiting factors. On the locked 100-item evaluation, plurality improved observed accuracy modestly, while confidence triage concentrated correct answers in high-ranked predictions.

  • Repeated sampling: 62/100 (62.0%; Wilson 95% CI [52.2, 70.9]) items were solved by four-sample plurality versus 58/100 (58.0%; Wilson 95% CI [48.2, 67.2]) by greedy decoding.The four-point gain equals a 6.9% relative improvement, but overlapping confidence intervals and a paired test that did not reject equality warrant describing it as observed, not established.
  • Repeated sampling: 79/100 items had at least one correct sampled answer, but plurality and the verifier recovered only 62/100, leaving a 17-point oracle gap.All 400 sampled answers were parseable; complexity alone did not improve selection, which must recover correct minority candidates without disrupting consensus.
  • Effort allocation: 44.4% was Opus 5’s Frontier-Bench peak at xhigh effort, while max effort fell to 43.3% despite consuming more compute.A community re-analysis placed the peak near medium effort at ∼53%, illustrating that maximum effort is not universally optimal.
  • Effort allocation: 2.6× and then 1.9× were the per-task cost multipliers for roughly four-point intelligence gains from Luna-max to Terra-max to Sol-max.Longer chains can wander on tasks with direct solution paths, so deployments should measure effort per workload and use its empirical optimum.
  • Model routing: 97.6% of the per-benchmark maximum was achieved by the best individual model across fourteen fully covered benchmarks, while Sol and Fable 5 together captured the full oracle gain.The fragmented frontier makes routing more effective than a single default model.

6 Discussion: how models get good at specific tasks

Models become strong at specific tasks through domain-targeted training, architecture, and inference orchestration, producing several frontier winners rather than one universal leader. This specialization makes evaluation, procurement, and confidence in reported gains depend on transfer, cost, provenance, and validation limits.

  • How models get good at specific tasks: Domain-specific reinforcement learning lets laboratories buy capability in chosen verticals through verifiable rewards such as unit tests, proof checkers, and task-completion signals.The passage links heavy reinforcement-learning investment to Moonshot’s agentic results and OpenAI’s terminal and agentic lead.
  • How models get good at specific tasks: BrowseComp 91.2 appeared as Kimi K3’s payoff from hybrid linear attention and expert routing designed for long-context agentic efficiency.The result was state of the art at release and appeared specifically on long-horizon benchmarks.
  • How models get good at specific tasks: 88.8% →91.9% shows GPT-5.6’s Ultra mode gaining Terminal-Bench points through multi-agent fan-out at higher cost.The passage presents inference orchestration, including confidence-based routing and selective additional compute or review, as part of capability.
  • Evaluation and limitations: ∼4× on ARC-AGI-3 did not prevent Opus 5 from statistically tying K3 and Fable 5 on held-out puzzles, illustrating benchmark optimization’s transfer limits.The passage identifies benchmark saturation and off-distribution failure as recurring risks when benchmarks become targets.
  • Implications for buyers: 20% of flagship token price matching last quarter’s flagship on most professional work shifts procurement from choosing one model to routing tasks across models and effort levels.The fragmented frontier replaces “which model is best?” with selecting combinations and effort settings for particular tasks.
  • Evaluation and limitations: 85.1% versus 82.6% for GPT-5.5 demonstrates how vendor reports and independent harnesses can diverge, while the 5.8× SWEbench trend is not a stable law.The routing oracle also knows benchmark identity, whereas production routers must infer task type from unlabeled traffic.
  • Evaluation and limitations: 16 development items and one 100-item GSM8K evaluation constrain the v1 intervention, whose four-point primary gain was not statistically significant (p = 0.481).The confidence model’s AUC and 94% top-half accuracy remain exploratory because labels informed its design and evaluation used the same 100 items.

7 Conclusion

The conclusion describes capability progress as a sequence of shifts across fine-tuning, few-shot scale, alignment, reasoning, and agency, alongside rapidly improving coding and falling capability costs. It identifies sample selection as a key system-level opportunity and provides a fully reproducible research archive.

  • Capability progress: Six-fold per year: agentic-coding solve odds continue rising, while a $1/M budget tier reproduces most of the professional capability at a fixed capability level.The passage contrasts roughly six-fold annual growth in solve odds with rapidly declining prices for fixed capability.
  • System-level gains: 58% to 62%: four-sample plurality improved frozen GSM8K exact match, while at least one sample was correct on 79% of items.The four-point difference remained uncertain (p = 0.481), whereas answer selection appeared to be the clearest next target.
  • System-level gains: 0.833 out-of-fold AUC: an exploratory logistic model used answer agreement and trace statistics, with 47 of its 50 highest-confidence predictions correct.The model’s signal supports confidence-based selection as a promising direction.
  • Reproducibility: The accompanying archive includes datasets, analysis scripts, locked manifests, inference traces, unit tests, figure generation, and the paper build script.Reproduction requires no proprietary access beyond live benchmark numbers, whose sources and retrieval dates are documented.
Loading 2608.13675v1…