PaperlayerView exact revision on arXiv

Source-linked AI summary

How Much Can Language Models Gain from Test-Time Computation?

Bangji Yang, Jingyuan Li, Jiajun Fan, Yi Evie Zhang, Ruihan Guo, Hongba Ma, Neil He, Chumeng Liang, Qinglong Zheng, Zhanghan Ni, Ge Liu

arXiv:2610.01110v1cs.LG

TL;DR

Prior test-time-scaling comparisons often cover one domain and rarely charge selection or verification to the same budget. SELF-POT evaluates fixed-budget protocols across static and agentic tasks, finding that returns vary by domain, selection rule, and failure handling.

  • Problem

    Prior comparisons often focus on one domain, rarely charge selection or verification calls to the same budget, and assess low-cost-versus-stronger-model comparisons only in narrow settings.

    Source: §1 · p. 2

  • Method

    SELF-POT evaluates fixed models across sealed domains using Direct, parallel sampling, and self-revision under a shared budget rule, while measuring candidate coverage, submission outcomes, revision transitions, and agent protocol completion.

    Source: §2.1 · p. 4, §2.2 · p. 4, §3.3 · p. 10

  • Results

    Returns vary by domain, selection rule, and failure handling; programming public-example selection raises correct submissions from 376 to 453 on fixed candidate pools.

    Source: §4 · p. 11, §6 · p. 13

  • Takeaways & Limitations

    The results distinguish missing or failed selections from poor decisions, pointing to improving the limiting stage before increasing its inference allowance.

    Source: §4 · p. 11

    Agent protocols with unexecuted plans cannot separate plan quality from execution reliability without additional controlled executions.

    Source: §4 · p. 12

Abstract

from arXiv · show

How much can test-time computation improve a language model, and at what cost? Test-time scaling is widely proposed as a substitute for larger models, but existing comparisons mostly evaluate one domain at a time and rarely charge selection to the budget. We introduce SELF-POT, a benchmark and evaluation framework that measures the test-time potential of a model across competition mathematics, competitive programming, and agentic workflows. SELF-POT separates candidate coverage from final accuracy on static tasks, tracks correctness transitions under revision, and measures protocol completion alongside task success in agentic environments. Under a unified budget rule, it compares Direct inference with parallel sampling and self-revision under fixed multiples of the Direct budget, and charges every model call, including selection and critique, in dollars. This design supports two kinds of comparison: the gain a model obtains from additional inference, and a lower-cost model with additional inference against a stronger model. Across five low-cost reasoning models on 350 sealed tasks, with Claude Opus 5.5 Direct as the reference, the returns depend on the domain, the selection rule, and failure handling. When we replay the retained programming candidate pools, public-example selection raises correct submissions from 376 to 453 of 500 scheduled cells while saving 12-49% of logical API cost across models, and simply retaining an available candidate when judging fails recovers 61 submissions at unchanged cost. On identical mathematics pools, judging with fallback yields 186 correct submissions versus 182 for voting, while voting saves 12-21% of logical API cost. These controlled replays show how selection and failure handling change the gains realized from the same generated candidates, and they quantify the marginal value of a model judge.

1 Introduction

The introduction frames test-time scaling as a way to spend inference computation instead of training, while asking when that helps across domains and at what cost. SELF-POT compares these choices under fixed budgets, charging for every model call and testing low-cost models against a stronger one-shot reference.

1 Introduction

  • Practitioners must choose among one attempt, parallel sampling with selection, and self-revision because their costs and available task feedback differ.Voting and public-example selection use no additional model tokens, model judging can cost as much as another answer, and flawed app-workflow actions cannot simply be discarded and rerun.Source: §1 · p. 2
  • Prior comparisons often focus on one domain, omit selection or verification costs, or compare weaker and stronger models only in narrow settings rather than across domains in dollars.The cited comparisons include matched-budget parallel and sequential methods [51], mostly mathematics evaluations [21], and narrower model comparisons.Source: §1 · p. 2
  • The study asks where extra computation improves accuracy, whether static-task opportunities are lost in generation or selection, and whether agent protocols complete their intended execution.It also asks whether scaled low-cost models can reach the accuracy of a stronger model answering once.Source: §1 · p. 2
  • SELF-POT holds the model fixed, charges selection and critique calls, and compares Direct with parallel sampling and self-revision at one, two, or four times the Direct budget.Parallel@2 uses voting in mathematics or public examples in programming; Parallel@4 uses joint selection, while Revise@4 critiques and revises.Source: §1 · pp. 2–3
  • The evaluation covers five low-cost reasoning models on 350 sealed tasks in competition mathematics, AtCoder programming, AppWorld workflows, and Terminal-Bench tasks, against Claude Opus 5.5 Direct.The mathematics sources are APEX and Omni-MATH [4] [20]; programming tasks come from LiveCodeBench [27], with AppWorld [50] and Terminal-Bench [38] covering agentic tasks.Source: §1 · p. 3
  • Programming public-example selection yields 453 correct submissions versus 376 for judging at 12–49% lower API cost; mathematics judging with fallback yields 186/292 versus 182/292 for voting.Programming joint selection loses 8.6 percentage points against Direct, but retaining the first available candidate when selection returns no choice turns this into a 3.6-point gain at unchanged cost.Source: §1 · p. 3

2 Methodology

SELF-POT measures how much additional inference a frozen model converts into task success, while charging every model call. It compares protocols under budget allowances across static and agent tasks, then analyzes where candidate selection or revision loses or gains accuracy.

2 Methodology

  • SELF-POT charges every model call and measures final task success under stated model, task, information, environment, harness, and budget conditions.Its measured rate estimates a configuration-specific test-time potential, not an intrinsic property of the model.Source: §2 · p. 3, §2.1 · p. 4

2.1 Measured quantity and budget

  • Each cell’s model calls share a protocol-scaled output-token cap, but realized token use and dollar cost can differ because protocols may stop early.The caps on calls, tools, and wall-clock time also scale with the protocol multiplier; reported accuracy is paired with realized use and cost, not a full response curve.Source: §2.1 · p. 4

2.2 Protocols

  • Static-task protocols compare Direct with parallel candidate selection and one self-revision iteration, while agent protocols select one plan to execute or review the ongoing trajectory.For static tasks, Parallel@2 uses a no-model-token rule, Parallel@4 adds joint selection, and Revise@4 critiques before revising; agent reviews continue from the current state because actions change the environment.Source: §2.2 · pp. 4–5

2.3 From candidate coverage to final accuracy

  • On static tasks, final success is candidate coverage multiplied by conversion, with covered failures split between no valid selection and an incorrect choice.Coverage uses hidden grades and is an offline ceiling, while replayed choice rules and fallback hold the generated candidate pools fixed and are not new equal-cost trials.Source: §2.3 · p. 5

2.4 Test set

  • The sealed test set spans mathematics, programming, workflows, and terminal tasks, but domain contrasts cannot isolate task type, difficulty, feedback, or protocol effects.Tasks were eligibility-checked, development items excluded, sampled within difficulty strata, and sealed before test-time calls.Source: §2.4 · p. 5, Table 1 · p. 6

2.5 Grading and cost accounting

  • Domain-specific graders determine correctness, absent or ungradable submissions count as incorrect, and selection, critique, planning, review, and execution calls count toward allowance and cost.Agent final-state grades remain authoritative even at resource limits; termination is reported separately, and costs are mean logical API costs over retained cells rather than invoices.Source: §2.5 · p. 6

3 Experiments

Results vary by domain and choice rule, and the deployment trade-off also depends on the model, selector, runtime, and price. Candidate coverage does not guarantee correct submissions, while fallback and task-appropriate selection can change the results from the same pools.

3.1 Setup

  • Only Flash and Qwen3-Next run revision because the other models lack configured context headroom for a full-budget critique.The experiment matrix has 68 configurations and 6,180 scheduled cells, with 6,165 final records.Source: §3.1 · p. 6, Table 2 · p. 6

3.2 RQ1: How gains vary across protocols and domains

  • Programming Parallel@2 gains 31 task–model outcomes with no losses, while Parallel@4 averages −8.6 pp; mathematics Parallel@4 averages +4.0 pp (95% interval [0.3, 7.7]).The programming result improves four models while Flash stays at 99/100; in mathematics, Parallel@4 raises accuracy for four of five models.Source: §3.2 · pp. 7–8, Table 4 · p. 8
  • Agent protocols do not raise panel means: workflow Parallel@2 and Parallel@4 fall 4.2 and 4.6 pp, and terminal Revise lowers Flash from 3/30 to 0/30.Periodic self-review also lowers Flash from 78 to 55 of 160 workflows, with 11 wins and 34 losses.Source: §3.2 · p. 7

3.3 RQ2: Converting candidate coverage into task success

  • Coverage can be lost during selection or submission: Qwen3-Next has correct mathematics candidates for 38 tasks but submits 30 correctly, all other outcomes being wrong choices.GLM-5 covers 88 programming tasks but submits 55 correctly; missing choices account for 27 of the 33 lost pools.Source: §3.3 · p. 8, Figure 4 · p. 9
  • On programming pools, fallback raises correct submissions from 376 to 437 of 500 at unchanged cost; public-example selection reaches 453/500 while saving 12–49% of API cost.Public-example selection converts 453 of 457 covered pools, and solves 94 versus fallback’s 78 among 98 pools containing both correct and incorrect candidates.Source: §3.3 · pp. 8–9
  • Mathematics judging with fallback yields 186 correct submissions on 292 retained pools versus 182 for voting; voting saves 12–21% of API cost.The observed accuracy difference has a paired interval spanning zero.Source: §3.3 · pp. 8–9, Figure 5 · p. 9

3.4 RQ3: When extra inference is competitive with a stronger model

  • Opus Direct leads in mathematics, workflows, and terminal tasks; in programming GPT-OSS Parallel@2 scores 97 against 93 at $0.052 against $0.028, while Flash Direct scores 99/100 at $0.013.The anchor reaches 99/100 with numpy and no new inference, making GPT-OSS’s programming advantage runtime-dependent.Source: §3.4 · p. 10, Figure 6 · p. 10
  • At half price, Flash judging ($0.088) and vote@3 ($0.072) join the expanded mathematics frontier, though both remain below Opus Direct.At peak rates, both are dominated by Opus Direct; judging buys two further correct answers.Source: §3.4 · pp. 10–11, Figure 6 · p. 10

4 Discussion

SELF-POT’s discussion argues that test-time gains depend on whether a correct candidate is generated, selected, and successfully submitted, as well as on the full configuration’s cost. The results support improving the limiting stage before spending more inference, while keeping agent execution and deployment conditions in view.

4 Discussion

  • Coverage, conversion, and revision transitions distinguish correct candidates from correct submissions and expose where test-time potential is lost.For GLM-5, 88 covered programming pools produce 55 correct submissions under the original judge, 78 with nonempty fallback, and 87 with public-example selection.Source: §4 · p. 11
  • 61 coding submissions are recovered by returning an available candidate when judging fails, without new inference.The discussion frames this as an avoidable failure-handling loss rather than a limit of candidate generation.Source: §4 · p. 11
  • 453 of 457 covered programming pools yield correct submissions under public-example selection, which avoids the judge’s API cost.A public-first judge gate yields 452 correct submissions, and mathematics judging with fallback yields four more correct submissions than voting on identical pools.Source: §4 · p. 11
  • A deployment comparison should include the model, protocol, harness, and price; Flash Direct remains the coding frontier after the replays.At half price, Flash vote@3 reaches 50/60 at $0.072/task and judging reaches 52/60 at $0.088; at peak rates both are dominated by Opus Direct.Source: §4 · p. 11
  • Evaluation reports should include the Direct operating point, actual resource use, selection outcomes, revision transitions, final grades, and termination status.The discussion also calls for a stronger Direct reference and runtime and price sensitivity analyses; agent plans that were not executed cannot provide oracle coverage estimates.Source: §4 · pp. 11–12

5 Related Work

Prior work studies test-time scaling across methods, domains, and cost measures, but the cited comparisons do not combine mathematics, programming, and agent tasks with charged selection calls and a stronger-model comparison. SELF-POT addresses that combination while using simple verifier-free forms of parallel and sequential scaling.

Test-time scaling

  • Parallel methods select among sampled candidates, while sequential methods revise an earlier attempt; SELF-POT runs the simplest verifier-free form of each family.Prior work reports that selection can limit gains and that self-correction without external feedback can fail to improve reasoning or turn correct answers wrong.Source: §5 · p. 12

Controlled comparisons

  • None of the cited controlled studies combines mathematics, programming, and agent tasks with charged selection calls and comparison against a stronger model.Table 5 identifies its scope as controlled studies of test-time scaling across mathematics, code, agent tasks, and other tasks, with parallel/sequential and judge-charging distinctions.Source: §5 · p. 12, Table 5 · p. 12

Cost-aware evaluation

  • Cost-aware evaluations use runtime, hardware, or dollars, while test-time scaling studies mostly count FLOPs; SELF-POT tests whether cheaper-model sampling is more economical.Other approaches include model cascades, routers, and adaptive stopping rules.Source: §5 · p. 13

6 Conclusion

SELF-POT measures how additional inference becomes task success and at what cost across sealed mathematics, programming, and agent tasks. Its conclusion highlights selection and failure handling as drivers of observed gains, while noting that agent outcomes require interface validation before attribution to planning.

6 Conclusion

  • The framework measures inference-to-success conversion across 350 sealed tasks in mathematics, programming, and agentic environments.The conclusion identifies producing a correct candidate versus successfully submitting it as a recurring distinction.Source: §6 · p. 13
  • Public-example selection raises correct programming submissions from 376 to 453, while output-preserving fallback recovers 61.In mathematics, judging with fallback adds four correct submissions over voting; for agents, interfaces require validation before outcomes can be attributed to planning.Source: §6 · p. 13
  • SELF-POT connects candidate coverage, executable selection rules, and cost to decisions about allocating the next unit of inference budget.The options include another candidate, a more reliable selection mechanism, or a different base model.Source: §6 · p. 13

A Budgets and caps

Static-task budgets scale each hard cap by the protocol’s multiplier, with nominal stage allowances specified for the configurations. The shared upper envelope does not make Parallel@4 and Revise@4 an identical-realized-cost mechanism comparison, and provider-specific limits can still truncate stages.

A Budgets and caps

  • Each domain has a unit budget B and hard caps, and a protocol at multiplier m receives m times every cap.Table 6 gives nominal static-task stage allowances; its suffix denotes a total budget multiplier, not a sample count.Source: A Budgets and caps · p. 20, Table 6 · p. 20
  • Provider limits can reduce individual allowances; Qwen3-Next’s revision is capped at B, giving at most 3B across its three stages.Provider-specific stages are not guaranteed to remain untruncated despite nominal headroom.Source: Table 6 · p. 20, A Budgets and caps · p. 20
  • Parallel@4 and Revise@4 share an upper budget envelope, but their accuracy difference does not isolate a mechanism at identical realized cost.Table 7 states that all Direct caps are multiplied by m for Parallel@2, Parallel@4, and Revise@4.Source: A Budgets and caps · p. 20, Table 7 · p. 20
  • One of Flash’s 60 mathematics Direct test cells was truncated, although the allowance exceeded its longest development-time Direct solution of 94,834 tokens.A single shell command is additionally limited to 1,800 seconds; task-native limits also apply.Source: A Budgets and caps · p. 20

B Test set construction

The test set is screened, stratified, and sealed before test-time inference. The protocol pseudocode allocates static-task budgets across generation, selection, critique, and revision, while agent protocols plan or review within a single episode.

B Test set construction

  • Tasks with executable references had to pass the reference and fail a blank submission; AppWorld and Terminal-Bench used their stated alternative eligibility checks.For Terminal-Bench, the Harbor oracle had to obtain reward 1 while the nop agent obtained reward 0; 62 of 66 tasks qualified.Source: B Test set construction · p. 21
  • Tasks were stratified by published difficulty, sampled without replacement with a fixed seed, and sealed before any test-time call.The sampling used seed 20260917, excluded development-seen tasks and held-out contests, and sealed the set on 2026-09-21.Source: B Test set construction · p. 21

C Protocol pseudocode

  • Static Parallel@2 draws two candidates and selects one without model tokens; Parallel@4 draws three and spends another B on a same-model judge.Revise@4 uses three stages: an initial answer, critique, and—unless the critique passes—a revision with allowance 2B.Source: C Protocol pseudocode · p. 22
  • Agent Parallel@k proposes k plans without tool calls, selects one, and executes it in one episode; Revise@4 periodically reviews the ongoing episode.Review occurs after every two executor turns, and the episode returns its final environment state.Source: C Protocol pseudocode · p. 22

D Protocol details

The static protocols differ in their selection, critique, and fallback rules, while agent protocols either execute one selected plan or review actions during a single episode. Several alternative reasoning and domain-specific protocols are outside the reported comparison.

Protocol details

  • Joint selection asks the model to verify three candidates and return a formatted choice; other replies count as format failures unless a single candidate letter is the last boxed expression.The latter parse route is recorded so the strict score remains recomputable.Source: D Protocol details · p. 23
  • Self-Refine uses one critique–revision iteration, but a failed critique or incomplete revision yields an empty submission rather than the initial answer.A PASS returns the initial solution; an unparseable verdict proceeds to revision.Source: D Protocol details · p. 23
  • Code Parallel@2 keeps the program that passes the most public examples, breaking ties toward the earlier candidate without using hidden tests or model tokens.Compile failures and empty programs receive a penalty below runnable programs that match no public examples.Source: D Protocol details · p. 23
  • Agent Parallel@k executes one selected plan, while Revise@4 appends periodic reviews to the history and continues from the current environment state.The program does not parse the review’s chosen option; the episode ends when the executor issues no further tool call.Source: D Protocol details · p. 23

Protocols not included

  • Longer-reasoning prompts, alternative revision allocations, and domain-specific methods were not included, so results do not compare those protocol variants.Direct’s high effort and generous allowance do not establish equivalence to all alternative prompts or provider effort settings.Source: D Protocol details · p. 23

E Grading and runtime

Static answers and agent outcomes are graded with domain-specific evaluators, while the agent harness prevents resetting episodes or choosing among executed worlds. Development observations informed the protocol design but do not establish that judging is preferable on held-out tasks.

E Grading and runtime

  • Static outputs use answer or execution graders, while agents are scored by native final-state evaluators; hidden answers and tests are reserved for offline grading.AppWorld task completion requires every native evaluator check to pass, and Terminal-Bench uses a binary native verifier reward.Source: E Grading and runtime · p. 24

F Agent harness

  • Agents cannot branch into cloned worlds, reset an episode, or select among multiple executed worlds, even though some individual actions may be undone.Each cell begins from its prescribed initial state, with no memory carried across tasks.Source: F Agent harness · p. 25
  • Terminal-Bench uses native Harbor containers and a single shell-command tool, so these scores are not comparable to the public leaderboard.A command is limited to 1,800 seconds; the limit terminates the process group and returns exit status 124.Source: F Agent harness · p. 25

G Development evidence behind the design

  • On 16 development problems, Direct scored 0.625, majority voting scored 0.562, and at least one of three samples was correct 81% of the time.On seven contested problems, joint verification recovered four of five cases with a correct candidate, while voting recovered one; these observations motivated design choices but do not establish held-out superiority.Source: G Development evidence behind the design · p. 26
  • The development observations are too limited to establish that judging is preferable on held-out tasks; the same-pool test analysis measures its marginal value separately.The development comparisons were small and served to motivate the original design.Source: G Development evidence behind the design · p. 26

H Accounting, failures, and changes after sealing

The diagnostics show that realized performance depends on accounting, failure handling, and converting recorded candidate coverage into submissions. Results also vary across mathematics, programming, and agent tasks, with some comparisons conditional on saved pools or specific runtime settings.

H Accounting, failures, and changes after sealing

  • The cost ledger reserves each stage cap before a call, keeps reservations for failed calls, and charges reused requests logically to each configuration.Its configuration means omit superseded infrastructure attempts and incomplete records, and do not reconstruct the actual invoice.Source: H Accounting, failures, and changes… · p. 27
  • Six of Opus 5.5’s seven code failures involved numpy missing from the frozen image; installing numpy made all six pass, raising the score from 0.930 to 0.990.No other model imported numpy.Source: H Accounting, failures, and changes… · p. 27
  • Post-sealing infrastructure changes affected tool-history serialization, tool handling, caching, or effective request caps, so nominal allowances did not ensure identical realized limits.The earlier-interface Qwen3-Next runs were retained as an implementation baseline rather than pooled into the final matrix.Source: H Accounting, failures, and changes… · pp. 27–28

J Offline diagnostic and statistical audit

  • The bootstrap resampled paired task outcomes within frozen strata, but models were fixed and the absence of repeated seeds prevented estimating within-task stochastic variability.The twelve panel intervals were pointwise descriptive intervals, not multiplicity-adjusted significance tests.Source: J Offline diagnostic and statistical… · p. 30
  • A fallback retaining the completed initial answer raised Flash programming revision accuracy from 98/100 to 99/100 at unchanged cost.Only one cell qualified; the replay did not apply to agent episodes, where actions may already have changed the environment.Source: J Offline diagnostic and statistical… · p. 30

J.1 Per-model diagnostics behind Section 3.3

  • On 292 available mathematics pools, self-selection submitted 185 correct answers and vote@3 submitted 182.The replay held candidate pools fixed; eight missing records remained failures in the primary 60-task denominators.Source: Table 11 · p. 31, §J.1 · p. 31
  • 81 observed covered programming pools failed to convert: 64 had no valid selection index and 17 selected an incorrect candidate.Of 61 coding outcomes lost against Direct, 56 had a retained record with no valid index, two had no final record, and three selected an incorrect nonempty candidate.Source: §J.1 · p. 32, Table 12 · p. 32
  • Flash’s workflow review reduced accuracy from 78/160 to 55/160, with 11 paired wins and 34 losses; separate episodes do not show individual trajectories being repaired into failure.Workflow parallelism also produced mixed model outcomes, and recorded failures were associated with absent planning messages or tool-call markup.Source: §J.1 · p. 32

J.2 Same-pool selection and failure-handling replay

On saved programming pools, public-example selection and fallback alter how often generated candidates become correct submissions, while changing API cost. Mathematics replays show a smaller selector difference, and deployment comparisons remain point estimates under specified price assumptions.

Programming replay

  • G calls the judge on 465 of 498 pools, only when public-score maxima tie, and omits judge cost on unique winners.Its choices match always judging with the same public-score guard.Source: §J.2 · p. 33
  • P3 yields 453/500 correct coding submissions (+6.8 pp vs Direct), while public-example selection saves 12–49% of logical API cost across models.The replay uses recorded Parallel@4 pools; P3 selects among all three candidates using public examples.Source: §J.2 · pp. 33–34, Table 13 · p. 33
  • Output-aware J+F recovers 61 submissions at unchanged logical cost, changing the gain against Direct from −8.6 to +3.6 pp.The fallback also handles a valid judge index that points to an empty candidate.Source: §J.2 · p. 33
  • Among 98 mixed-correctness pools, public selection returns 94 correct submissions versus 78 for fallback and 61 for judging.Its 16 additional correct choices over fallback account for its entire additional gain in these pools.Source: §J.2 · p. 34, Table 15 · p. 34

Replay scope

  • These are fixed-pool accounting counterfactuals: generation costs remain charged, and replay variants are not fresh end-to-end trials at matched expenditure.Fallback retains judge-stage spending; policies without judging remove those calls, including continuations.Source: §J.2 · p. 34

Mathematics replay

  • Fallback raises mathematics judging to 186/292 correct submissions versus 182/292 for voting; voting saves 12.3–21.1% of logical API cost.The fallback recovers one MiniMax answer, increasing its count from 27 to 28; removing judge-stage calls saves the stated cost.Source: §J.2 · p. 34, Table 16 · p. 35

Mathematics grading sensitivity

  • Under Compass grading, judging with fallback scores 194 correct versus 189 for voting, but the interval for their difference includes zero.The pointwise 95% interval is [−1.04, 4.79] pp; this sensitivity analysis does not replace the frozen primary scores or constitute human adjudication.Source: §J.2 · p. 35

Deployment comparison

  • At half the Flash price, vote@3 joins the mathematics frontier at 50/60 and $0.0722/task, below judging’s 52/60 and $0.0882.These point-estimate API-cost comparisons omit local execution expense and do not establish statistically certified dominance.Source: §J.2 · p. 35

K Full results

The full-results matrix compares each cheap model’s protocol accuracy and cost with its own Direct run and a Claude Opus 5.5 Direct reference. Its code score includes a sandbox condition for some Opus programs.

Full results

  • Table 18 adds paired wins–losses against the same model’s Direct run and relative cost for every configuration.The matrix marks exact two-sided sign-test thresholds without correction and identifies cells that never finished as incorrect.Source: K Full results · p. 37, Table 18 · p. 37
  • The frozen protocol matrix reports each cheap model’s accuracy by protocol, same-task wins and losses against Direct, and cost relative to Direct.Claude Opus 5.5 answering once is the expensive anchor; some configurations were excluded or not run, and some plan rows reflect format compliance.Source: Table 18 · p. 37
  • Opus 5.5 scores 0.990 on code when numpy is available in the grading sandbox; six of its programs import it.Costs average retained records at configured rates, and missing records have no imputed cost.Source: K Full results · p. 37
Loading 2610.01110v1…