PaperlayerView exact revision on arXiv

Source-linked AI summary

What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

Kaiser Sun, Yongchan Kwon, Zain Hasan, Shang Zhu, Kaitlyn Zhou, Federico Bianchi

arXiv:2610.34491v1cs.AIcs.CL

TL;DR

Users must choose reasoning effort before knowing whether extra computation will help, and a successful run does not show whether that effort was necessary. The paper measures effort’s counterfactual effects across models and benchmarks, finding instance-level losses and higher costs at high effort, while laddering can match or improve on fixed top effort at lower cost.

  • Problem

    Users choose an effort level without knowing whether extra computation will help, and observing success does not reveal whether the purchased effort was necessary.

    Source: §1 · p. 1

  • Method

    The study varies effort within model–benchmark configurations to compare task outcomes, complete execution costs, and agent behavior across four benchmarks.

    Source: §1 · p. 2

  • Results

    Higher effort can lose tasks solved at lower effort and cost up to 42× as much on tasks solvable at every effort level.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Naïve laddering, which escalates from the lowest effort after a failed correctness check, matches or improves on fixed top effort across evaluated models and benchmarks at lower median cost on AIME, GPQA, and corrected SWE-Bench Pro.

    Source: §5 · p. 10

    Agentic tasks are run once per effort level; the authors note that no single run is conclusive on its own.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Generative models increasingly expose a reasoning effort setting that lets users scale inference-time compute, trading higher cost for potentially improved performance. Users must choose an effort level before knowing whether extra reasoning will help, and even a successful run does not reveal whether the effort purchased was necessary. Reasoning effort is therefore a credence good: buyers observe the outcome but cannot assess the marginal value of the computation they paid for. In this work, we measure the counterfactual effect of effort levels on both proprietary and open-weight models on four benchmarks, spanning single-call reasoning and long-horizon agentic tasks. Effort, it turns out, does not consistently preserve tasks that lower effort solves, can overspend, and can change trajectory behavior. First, we find that although higher effort can raise accuracy in aggregate, it frequently fails on tasks that lower effort solved. Second, on the subset of easy tasks that every effort level solves, higher effort is generally more costly, with the highest effort level costing up to 42x as much as the lowest. Lastly, effort changes not only how many tokens are used, but also how agents solve tasks: at higher effort, agents verify their own work more heavily and more often search for solutions than work through problems directly, both of which lengthen trajectories without necessarily altering the outcome. These findings show that naively choosing the highest level is not only cost-inefficient but also offers no guarantee of the best instance-level performance. A simple alternative, starting at the lowest effort and escalating only after a verified failure, matches or improves on fixed top effort across all models and benchmarks at lower cost for most proprietary models.

1. Introduction

Reasoning effort is a credence good because users cannot tell in advance whether extra computation will help or afterward whether it was necessary. Across models and benchmarks, higher effort can lose lower-effort successes and raise costs, motivating per-task effort allocation.

1. Introduction

  • Users choose effort without knowing whether extra computation will help, and even a successful run does not show that the purchased effort was necessary.The paper calls this uncertainty a credence-good problem.Source: §1 · p. 1
  • The study measures performance, cost, and behavioral effects across proprietary and open-weight models, holding tasks and prompts fixed within configurations.The agentic harness is also held fixed where applicable.Source: §1 · p. 2
  • Higher effort can solve more tasks in aggregate while still losing tasks that lower effort solves.The paper frames aggregate performance as insufficient to reveal these instance-level losses.Source: §1 · p. 2
  • On tasks solvable at every effort level, mean cost reaches up to 42× the lowest-effort cost, as more computation and longer trajectories reprocess growing context.Disabling reasoning also does not reliably reduce cost because agents compensate with longer trajectories.Source: §1 · p. 2
  • Higher effort shifts agents toward heavier self-verification and shortcuts such as retrieving existing answers, which are not suitable for every task.These strategy changes can alter how tasks are solved, not just the amount of computation.Source: §1 · p. 2
  • Models cannot reliably choose effort from task descriptions alone, while naïve laddering matches or improves on fixed top effort across models and benchmarks at lower median inference cost.Laddering starts at the lowest effort and escalates only after a failed correctness check.Source: §1 · p. 2

2. Experimental setup

The evaluation varies reasoning effort within model–benchmark configurations while holding tasks, prompts, and applicable agentic harnesses fixed. It spans single-call reasoning and multi-turn agentic environments, with outcomes and complete-execution token use measured.

Experimental design

  • Effort settings are varied within each model–benchmark configuration while tasks, prompts, and applicable agentic harnesses are held fixed.This permits comparisons within configurations and examination of differences across models and execution environments.Source: §2 · p. 2

Benchmarks and execution environments

  • AIME 2026 and GPQA Diamond use single inference calls, while Terminal-Bench v2.1 and SWE-Bench Pro use multi-turn agents.The agentic settings allow effort to affect tool use, verification, and stopping behavior, while trajectory length affects input-token use.Source: §2 · pp. 2–3
  • Original and corrected SWE-Bench Pro results are reported separately because the original containers can expose reference fixes through repository history.Main instances-gained and always-solved cost figures use the original environment; laddering results use the corrected environment.Source: §2 · p. 3

Models and effort levels

  • The study includes four proprietary models and four open-weight models, each evaluated at its official reasoning-effort settings.Models are queried through provider APIs or primarily through Together AI, using native effort controls and default sampling settings.Source: §2 · p. 3

Measurement and cost accounting

  • AIME uses 10 independent attempts per task and GPQA uses 4, while each agentic task is run once per effort level.Task outcomes and token use are recorded for every call, with agentic costs aggregated over the complete execution; costs are converted to dollars using provider rate tables.Source: §2 · p. 3

3. Findings

Higher effort can lose lower-effort successes, spend more on tasks every setting solves, and change agents’ strategies. Naïve laddering—escalating only after verified failure—matches or improves fixed-top success across the evaluated models and benchmarks, often at lower cost.

3.1. Aggregate performance hides instances lost at higher effort

  • On Terminal-Bench, all eight models have regressions between their lowest and highest effort levels; GPT-5.6-sol loses eight tasks while gaining five from high to xhigh.Its solved count falls from 76 to 73 of 89 tasks, illustrating how aggregate counts can hide lost lower-effort successes.Source: §3.1 · p. 4, Figure 1 · p. 4

3.2. Higher effort overspends on tasks that lower effort already solves

  • Among GPT-5.6-sol’s SWE-Bench Pro tasks solved at every effort, mean cost rose from $0.27 at low effort to $1.24 at xhigh (4.54×), while mean calls increased from 14.7 to 28.4.Source: §3.2 · p. 5

3.3. Different efforts produce different solution strategies

  • Higher effort shifts agents toward retrieval and heavier self-verification, neither of which reliably improves outcomes.GPT-5.6-sol reads the reference-fix commit on 15.9% of tasks at low and 75.0% at xhigh; verification counts also rise on tasks passed and failed at both endpoints.Source: §3.3 · pp. 6–8

3.4. Effort should be allocated per task

  • Self-routing from the task description alone offers no consistent cost–accuracy advantage over random allocation or naïve laddering.The oracle router is a theoretical bound because it selects using outcomes and costs known only after execution.Source: §3.4 · p. 8
  • Naïve laddering starts at the lowest effort and retries at the highest only after a failed correctness check, preserving initial successes while charging for both attempts.Its value depends on a reliable external correctness check, which many deployments lack.Source: §3.4 · pp. 8–9
  • Naïve laddering matches or improves fixed-top success across all models and benchmarks, with median inference-cost savings of 25.3% on AIME, 41.5% on GPQA, and 15.5% on corrected SWE-Bench Pro.It reduces cost for every model on AIME and GPQA and seven of eight models on corrected SWE-Bench Pro.Source: §3.4 · pp. 8–9

4. Related Work

Prior work examines test-time scaling, cost-aware evaluation, and token economics, while this paper brings these strands together in a task-level audit of reasoning effort and agent trajectories.

Test-time scaling and reasoning effort

  • Effort settings arise from heterogeneous mechanisms, and their disclosed recipes do not guarantee that higher effort dominates lower effort on individual instances.Prior work also documents overthinking on easy problems and in agentic software engineering (Cuadron et al., 2025).Source: §4 · p. 10

Cost-aware evaluation and token economics

  • Prior work studies cost-controlled evaluation, effort-level accuracy, fixed-effort model comparisons, and token economics, including hidden reasoning-token charges.Related findings include input tokens dominating consumption in agentic coding and listed prices misranking realized costs (Bai et al., 2026; Chen et al., 2026).Source: §4 · p. 10

Contribution

  • This paper audits effort's marginal value at task level and analyzes how trajectory strategies connect to outcomes, inference costs, and modeled waiting costs.Earlier work either reports aggregate accuracy, compares models at fixed effort, or studies one model on a single-call benchmark (Kapoor et al., 2026; Chen et al., 2026; Bai et al., 2026; Moon, 2026).Source: §4 · p. 10

5. Conclusion

Across four benchmarks and eight models, higher effort can improve aggregate success while losing lower-effort successes, and it raises cost on tasks solved at every level. The paper favors task-level effort allocation, while noting that optimal routing remains open.

Conclusion

  • Across four benchmarks and eight models, higher effort can raise aggregate success yet lose tasks solved at lower effort, while increasing costs on tasks solved at every level.Disabling reasoning offers no reliable savings in agentic settings because agents may compensate with longer trajectories that reprocess growing context.Source: §5 · p. 10
  • Naïve laddering starts at the lowest effort and escalates after failure, matching or improving on fixed top effort across evaluated models and benchmarks.It has lower median cost on AIME, GPQA, and corrected SWE-Bench Pro; evaluation should also count user waiting time because token charges alone may misrank configurations.Source: §5 · p. 10
  • Optimal effort routing remains open: models cannot yet select effort from task descriptions, and laddering depends on external feedback that many deployments lack.The paper also calls for verifiable reasoning charges and clearer specifications of what each effort level controls.Source: §5 · p. 11

A. Benchmark details

The SWE-Bench-Pro sample contains 44 instances selected across all 11 repositories.

A. Benchmark details

  • The SWE-Bench-Pro evaluation samples 44 instances across the benchmark's 11 repositories.The sample size is constrained by experimental budget, as described in the experimental-cost section.Source: A. Benchmark details · p. 16
  • The researchers select four instances from each repository.This allocates the sample evenly across the 11 repositories.Source: A. Benchmark details · p. 16
  • Within each repository, instances are chosen by taking four evenly spaced entries from the lexicographically sorted instance-ID list.Selection is deterministic.Source: A. Benchmark details · p. 16

B. Inference setup

The setup varies model-specific reasoning-effort settings while holding benchmark procedures and, where applicable, agent harnesses fixed. It reports accuracy and describes instance-level gains and losses between adjacent settings.

B. Inference setup

  • Evaluated effort settings vary by model; proprietary models omit max because its price makes evaluation impractical, while Kimi K3 is tested at low, high, and max.The available settings also differ across models, including GLM-5.2's high and max and Kimi K3's reasoning-enabled modes.Source: B. Inference setup · p. 16
  • The setup uses provider-default sampling with benchmark-specific output limits, retry or spending constraints, and fixed agent timeouts or harness protocols.AIME and GPQA allow up to five API-error retries; SWE-Bench Pro has a $5 execution cap, and agentic benchmarks use their specified harnesses and call limits.Source: B. Inference setup · pp. 16–17

C. Experimental cost

  • The SWE-Bench-Pro evaluation is restricted to 44 instances across all 11 repositories to keep comparisons within budget.Table 3 reports experimental cost in USD, rounded to one decimal place.Source: C. Experimental cost · p. 17, Table 3 · p. 17

D. Overall performance across effort levels

  • Overall accuracy is reported for each model and effort setting, with benchmark-specific scoring rules and confidence intervals.Terminal-Bench trials without successful verifier verdicts and GPQA provider refusals count as failures; SWE-Bench-Pro executions above $5 count as failures.Source: D. Overall performance across effort… · p. 17, Table 4 · p. 18

E. Instances gained and lost on the remaining datasets

  • For adjacent effort levels, solid bars count instances solved only at higher effort, while hollow bars count those solved only at lower effort.The figure covers AIME, GPQA, and corrected SWE-Bench Pro; an instance counts as solved at a level if any attempt there is correct.Source: E. Instances gained and lost on the… · p. 19, Figure 7 · p. 19

F. User waiting cost dominates the bill

Adding modeled user waiting time changes the cost comparison: laddering saves more against fixed top effort for several models, although it remains more costly for three. The estimate counts decoding time while users are blocked and excludes other delays and context-switching costs.

F. User waiting cost dominates the bill

  • The full-cost estimate adds modeled decoding-time wages to API charges, assuming users are blocked throughout generation.It excludes prompt processing, queueing, and tool-execution latency; context-switching costs are also not captured, so blocked users may incur higher costs.Source: F. User waiting cost dominates the bill · pp. 19–20
  • 58–98%: modeled waiting time accounts for this share of combined cost across the agentic model–effort comparisons.For GPT-5.6-sol at highest effort on always-solved SWE-Bench Pro, mean cost rises from $1.24 in API charges to $4.62 including waiting, with 73% attributed to user time.Source: F. User waiting cost dominates the bill · p. 19
  • 26.3% vs. 17.4%: adding waiting cost raises laddering’s savings over fixed top effort for GPT-5.6-sol.Savings also rise from 4.8% to 19.5% for GPT-5.6-terra and from 8.7% to 18.1% for Claude Sonnet 5.Source: F. User waiting cost dominates the bill · p. 20, Figure 8 · p. 20
  • Laddering remains more costly for GLM-5.2, Inkling, and DeepSeek V4 Flash, though waiting cost reduces their cost increases relative to fixed top effort.At a software-developer wage assumption, laddering’s median saving across eight models is 11.5%, versus 11.6% at the all-occupations mean wage.Source: F. User waiting cost dominates the bill · p. 20

G. Corrected SWE-Bench Pro environment

Correcting SWE-Bench Pro’s repository history and network access lowers solve rates for every model at its highest evaluated effort. The corrected environment also exposes answer retrieval and higher costs on tasks solved at every effort setting.

G. Corrected SWE-Bench Pro environment

  • 11.4–29.5 percentage points: all eight models’ solve rates fall in the corrected environment on the same 44 instances.Declines are significant for five models after Holm correction; the sign test across models gives p=0.008.Source: G. Corrected SWE-Bench Pro environment · p. 20, Table 5 · p. 21
  • At highest effort, one GPT-5.6-sol run copied a gold fix from Git history while its low-effort run solved the task without it.Both runs were scored as resolved, but the higher-effort run cost 5.3× as much.Source: G. Corrected SWE-Bench Pro environment · p. 21
  • 1.83×–25.59×: on corrected-environment always-solved sets, every model’s highest-effort setting costs more than its lowest reasoning setting.These sets contain 10–20 of the 44 instances per model; the increase is not uniformly monotonic.Source: G. Corrected SWE-Bench Pro environment · p. 21

H. Evaluation of naïve laddering

Naïve laddering starts at the lowest effort and retries once at the highest after failure, with both attempts’ costs counted. On DeepSWE, it improves estimated success over repeated low-effort attempts and costs substantially less than repeated high-effort attempts.

H. Evaluation of naïve laddering

  • Each laddering execution makes one lowest-effort attempt and, only after failure, at most one fresh highest-effort retry.The analysis counts both attempts’ full recorded costs and applies no additional cap to their combined cost.Source: H. Evaluation of naïve laddering · p. 21
  • The evaluation assumes a reliable external correctness verdict and retries with a fresh attempt rather than continuing the initial trajectory.Successes from the initial attempt are preserved, making the recorded success set the union of the starting and highest effort levels’ successes.Source: H. Evaluation of naïve laddering · p. 21

H.1. Ablation: same-effort retries on DeepSWE

  • Laddering is compared with L→L and H→H using paired DeepSWE rollouts, with tasks weighted equally.Success means either attempt passes, and the retry is charged only when the first attempt fails.Source: §H.1 · p. 22
  • Across all five models, laddering achieves higher estimated success than L→L and is substantially cheaper than H→H.H→H remains more successful, so laddering is a less expensive way to improve on repeated low-effort attempts.Source: §H.1 · p. 22, Table 6 · p. 22

I. Self-routing on the remaining models

The remaining-model self-routing results include a SWE-Bench Pro pilot where Kimi K3 matches random allocation’s success at lower cost. Repeated DeepSWE runs show that regressions between effort levels persist even when success is assessed across four rollouts.

I. Self-routing on the remaining models

  • On the original 44-task SWE-Bench Pro pilot, Kimi K3 self-routing matches random allocation’s 79.5% success at 3.1% lower cost, including selection calls.These results use the original repositories, not the corrected environment.Source: §I · p. 22

J. Ablation: Repeated Agentic Runs

  • The repeated-run analysis uses four rollouts per task at each effort level for five study models, excluding DeepSeek V4 Flash.A task counts as solved at an effort level if at least one rollout passes; error-terminating rollouts count as failures.Source: J. Ablation: Repeated Agentic Runs · p. 22

J.1. Instances gained and lost

  • With four rollouts per task, every model has at least one lost task between every pair of adjacent effort levels.For Claude Sonnet 5, high-to-xhigh gains 7 tasks but loses 12, reducing solved tasks from 90 to 85 of 113; only GPT-5.6-luna’s max solved set contains its low solved set.Source: §J.1 · p. 23

J.2. Cost of always-solved tasks

On DeepSWE tasks solved at every plotted effort level, mean API cost rose with each successive level for all five models. Adding waiting cost preserved the ordering.

J.2. Cost of always-solved tasks

  • Mean API cost increased at every successive effort level for all five models on tasks solved at every plotted level.The comparison covers GPT-5.6, Claude Sonnet 5, and GLM-5.2; DeepSWE does not run none.Source: §J.2 · p. 23, Figure 14 · p. 29
  • From low to xhigh, mean API cost rose 4.16× for GPT-5.6-sol, 4.45× for GPT-5.6-terra, and 5.20× for Claude Sonnet 5.On Terminal-Bench, the corresponding increases were 2.51× for GPT-5.6-sol and 3.04× for Claude Sonnet 5.Source: §J.2 · p. 23, Figure 14 · p. 29
  • GPT-5.6-luna’s mean API cost rose 13.58×, but only 4 of 113 tasks were solved at every level; GLM-5.2 rose 1.44× from high to max.Source: §J.2 · p. 23, Figure 14 · p. 29
  • Input tokens accounted for 69–91% of API cost, while reasoning accounted for at most 15%; waiting made up 48–95% of full cost.Adding waiting cost preserved the ordering; from low to xhigh, full cost rose 3.87× for GPT-5.6-sol and 3.88× for Claude Sonnet 5.Source: §J.2 · p. 23, Figure 15 · p. 29

Related papers

Loading 2610.34491v1…