PaperlayerView exact revision on arXiv

Source-linked AI summary

Budget Boundary Effects in Test-Time Mathematical Reasoning

Guilin Zhang, Ziqi Tan, Wulan Guo, Kai Zhao, Hongyun Yang, Mei Luo, Qi Ning, Feng Yang

arXiv:2609.38699v1cs.AI

TL;DR

A cumulative token cap can interrupt mathematical reasoning, leaving controllers to stop at the cap or allow the current attempt to finish. This paper compares those rules in paired offline replays of public traces, finding that short-cap gains largely replace abstentions while realized cost and selector behavior complicate comparisons.

  • Problem

    When a cumulative cap falls inside an attempt, it is unclear how strict stopping versus allowing completion changes eligible answers and the returned decision.

    Source: §1 · p. 1

  • Method

    The authors replay 19,200 public traces under paired strict and advisory stopping rules, using fixed orders and a label-blind selector on completed eligible answers.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    At 4k, advisory gains mostly replace abstentions with correct answers; cross-cap accuracy comparisons vary with realized cost, and a log-probability selector can lose accuracy as coverage rises.

    Source: §4 · pp. 3–4

  • Takeaways & Limitations

    Budget curves should report the cap, realized cost, eligible candidates, stopping rule, and selector information together.

    Source: §5 · p. 4

    The evidence covers one model’s two archives and a completed-only sequential controller, and does not establish an online matched-compute advantage.

    Source: §5 · p. 4

Abstract

from arXiv · show

A cumulative token cap can fall inside a mathematical derivation, forcing a test-time controller to choose between stopping at the cap (strict) and allowing the current attempt to finish (advisory). We measure this boundary choice with paired offline replays of 19,200 public traces: 120 AIME, BrUMO and HMMT problems and two archive configurations of one model. Candidate order and a 16-attempt cap are fixed, and answer selection is blind to reference answers and correctness labels. Three findings emerge. First, at the 4k cap, most advisory accuracy gains replace abstention with a correct answer; strict stopping pays for an unfinished prefix that the completed-only selector cannot use. Second, comparisons along realized cost differ from same-cap comparisons: advisory 4k in low has higher accuracy than strict 8k at comparable mean completion cost, while in high its observed accuracy is 0.42 points below strict 32k using 59% of its mean tokens. These aggregate comparisons do not establish equal-compute superiority or accuracy equivalence. Third, increased candidate coverage does not guarantee higher answer accuracy: a log-probability selector loses accuracy while coverage rises, including after a source-grade consistency repair. Same-cap majority-accuracy differences shrink below 1.3 percentage points at 32k. Budget curves should jointly state the cap, realized cost, eligible candidates, stopping rule and selector information.

1 Introduction

The paper asks how a cumulative token cap affects a mathematical solver when it lands inside an attempt. It audits strict stopping versus allowing that attempt to finish, separating answer availability, selection, and evaluation.

1 Introduction

  • A cap inside an attempt creates a practical accounting question because mathematical solvers spend budget on intermediate reasoning, sampled solutions, or tree search.The paper situates this question alongside work on compute allocation and length control.Source: §1 · p. 1
  • The audit separates three effects: abstention drives short-budget gains, realized cost changes cross-budget comparisons, and coverage can rise while selected-answer accuracy falls.The study holds archived attempts and the selector fixed while varying the stopping rule.Source: §1 · pp. 1–2
  • The audit varies how interrupted work is charged and separates candidate coverage, label-blind selection, and evaluation.Its paired evaluation fixes the archived bank and order, then applies eligibility and selection separately to each pool.Source: §1 · p. 2, Figure 1 · p. 2

2 A paired boundary audit

The paired boundary audit replays a fixed attempt order under strict and advisory stopping, then compares eligible answer pools using a label-blind selector. It distinguishes candidate availability from the selector’s returned decision.

2 A paired boundary audit

  • The replay uses complete archives to reconstruct costs, but its simulated controller launches attempts using only tokens spent so far, not eventual length or correctness.It allows at most K attempts and follows one predetermined order per problem.Source: §2 · p. 2
  • A strict boundary-crossing attempt contributes consumed prefix tokens but no answer; advisory stopping launches an attempt before the cap and lets it finish.Only records marked stop with a valid parsed answer enter either selector; the completed-only rule does not infer answers from truncated prefixes.Source: §2 · p. 2
  • The primary selector takes the plurality of normalized boxed answers, breaks ties by first occurrence, and cannot access references or correctness labels.An empty pool counts as an incorrect abstention; archived correctness is read only afterward for evaluation.Source: §2 · p. 2
  • Coverage measures whether a pool contains any archived-correct answer, while paired accuracy measures the returned decision; the advisory pool adds at most one attempt.Coverage therefore cannot decrease, but the majority decision can improve or worsen.Source: §2 · p. 3

3 Data and experimental design

The experiment reuses public traces from 120 competition problems and two archive configurations, with no new model outputs. It fixes the attempt cap and samples predetermined candidate orders for paired policy comparisons.

3 Data and experimental design

  • The study reuses 19,200 existing attempts from 120 AIME, BrUMO, and HMMT problems across gpt-oss-20b_low and gpt-oss-20b_high archives.Each configuration has 80 archived seeds per problem; low/high are treated as archive labels, not verified causal interventions on API effort.Source: §3 · p. 3
  • The primary analysis fixes K = 16 and uses caps of 4k, 8k, 16k, and 32k completion tokens, with 20 predetermined orders per problem.The same order is seen by both policies; sensitivity analyses use canonical order and K values of 8 or 32.Source: §3 · p. 3
  • Strict cost includes discarded partial-attempt tokens, whereas advisory cost includes the final returned attempt; prompt tokens are accounted separately under a no-cache assumption.Invalid outputs and parser failures retain their cost and remain in the problem denominator.Source: §3 · p. 3

4 Results and controls

Advisory stopping improves majority accuracy most at the 4k cap, largely by replacing abstentions with correct answers; realized-cost comparisons and selector sensitivity qualify that result. At 32k, same-cap accuracy differences are small, and added candidate coverage does not always improve selected answers.

4 Results and controls

  • All 43.04 pp of high-archive improvement at 4k replaces abstention with a correct answer, while abstention falls from 80.54% to 32.00%.Advisory returns exactly one attempt in 80.50% of these replays, with a mean eligible pool of 0.94 answers; in low, abstention-to-correct contributes 11.54 pp.Source: §4 · p. 3
  • In low, advisory 4k reaches 50.13% at 7,485 mean tokens versus strict 8k at 46.96% and 7,940 tokens; in high, advisory 4k is 0.42 pp below strict 32k while using 58.77% of its mean cost.The high comparison does not establish accuracy equivalence, and these comparisons do not hold each problem’s compute fixed.Source: §4 · pp. 3–4
  • At 4k, advisory majority accuracy rises from 37.25% to 50.13% in low and from 18.54% to 61.58% in high; at 32k the changes shrink to 0.63 and 1.25 pp.Table 1 reports the paired boundary audit; the low-archive 4k gain is concentrated in AIME24/25 (25.00/23.33 pp), versus BrUMO/HMMT (1.83/1.33 pp).Source: §4 · p. 3, Table 1 · p. 4
  • At 32k in low, maximum mean log-probability loses 0.71 pp while coverage rises by 0.88 pp; all 31 harmed outcomes persist after evaluation-only grade repair.The repair changes paired majority effects by at most 0.08 pp in low and 0.33 pp in high; selectors remain unchanged.Source: §4 · p. 4

5 Scope and implications

The audit isolates a fixed-order accounting choice rather than comparing adaptive policies, and its scope is one model’s two archives under completed-only sequential control. The results do not establish an online matched-compute advantage.

5 Scope and implications

  • Length control, dynamic early exit, and answering without extended thinking change generation or the terminal rule and require their own trajectories.The paper distinguishes completing an attempt, which addresses answer availability, from improving the reducer, which addresses selection.Source: §5 · p. 4
  • The replay does not compare adaptive early stopping, optimized sample allocation, or stochastic backtracking policies.Those approaches alter generation or allocation; the fixed-order replay isolates an accounting choice they must also specify.Source: §5 · p. 4
  • The estimates cover one model’s two archives and a completed-only sequential controller with shared order and eligibility.Normalization can split equivalent expressions, and consistency checks only partly address source-grade errors.Source: §5 · p. 4
  • The findings neither rank adaptive agents nor establish an online matched-compute advantage.Budget curves should report the cap, realized cost, attempted and eligible counts, stopping rule, and selector information together.Source: §5 · p. 4

A Boundary identities and reproducibility

The appendix formalizes the paired boundary comparison and documents the replay inputs and evaluation checks. Its length-stratified findings are descriptive, and the high archive retains a truncation-related abstention floor.

A Boundary identities and reproducibility

  • Advisory adds at most one boundary attempt to the strict pool, so coverage cannot fall, but majority accuracy can improve or worsen.A boundary-crossing strict attempt is charged to the ledger without contributing an answer; only completed, valid parsed answers enter either selector.Source: A Boundary identities and… · p. 6
  • The paired majority change is the probability of strict-wrong/advisory-correct minus strict-correct/advisory-wrong, while coverage change counts only strict-empty/advisory-nonempty cases.The nested pools make the coverage-loss event impossible.Source: A Boundary identities and… · p. 6
  • The replay uses public Scorio Trace inputs, with 2,400 records per configuration and competition, comprising 30 questions and 80 seeds.The pinned revision and parsing, finish-status, control, and bootstrap settings are recorded; replaying archived inputs does not generate new reasoning attempts.Source: A Boundary identities and… · p. 6
  • Table 2 audits 2,400 attempts per row; its no-box and length columns overlap, and length records can remain costly while ineligible.Normalization does not simplify algebraic expressions or compare candidates against references.Source: Table 2 · p. 7, A Boundary identities and… · p. 7
  • At every tested budget, high-archive advisory abstention is 32.00% because 768 of 2,400 pairs begin with an ineligible 32,768-token length-capped response.That inherited generation cap prevents a second attempt under the eligibility rule; it is not an estimate of irreducible model failure.Source: A Boundary identities and… · p. 7
  • Table 3 compares primary source-grade paired changes with exploratory formatting-repaired grades; the repair adjusts evaluation labels without changing selection.It resolves the detected label-conflict groups but does not claim complete symbolic grading or repair false-positive source labels.Source: Table 3 · p. 7, A Boundary identities and… · p. 7
  • The length halves are descriptive conditional groups, not causal effects of making a solution longer.Questions are divided by median candidate cost, and competition or difficulty composition may differ between halves.Source: A Boundary identities and… · p. 7, Figure 3 · p. 8
  • At 4k in high, advisory gains 31.50 pp in the longer half and 54.58 pp in the shorter half; in low, gains are 24.08 pp and 1.67 pp, respectively.The longer high-archive half spends an extra 23,697 completion tokens on average, versus 5,761 in the shorter half; Figure 3 reports descriptive length-stratum comparisons.Source: A Boundary identities and… · p. 8, Figure 3 · p. 8

B Exploratory comparisons along realized cost

Comparisons by realized cost yield different conclusions from same-cap comparisons, but aggregate costs do not establish matched-compute performance. Interpolation suggests advisory 4k accuracy exceeds the strict curve at its mean cost, subject to assumptions about curve shape.

B Exploratory comparisons along realized cost

  • Table 4 compares advisory at a fixed nominal 4k cap with strict policies at varying caps using mean realized completion cost and archived-grade accuracy.These are different-cost comparisons.Source: Table 4 · p. 8
  • Mean-cost comparisons leave per-problem compute allocation uncontrolled, and the high advisory-4k/strict-32k accuracy interval does not establish equivalence.The observed cost ratio is 0.588 [0.539, 0.640].Source: B Exploratory comparisons along… · p. 8
  • 4.28 pp in low and 11.83 pp in high are advisory-minus-interpolated strict accuracy differences at advisory 4k’s mean cost.The strict endpoints are 4k–8k in low and 16k–32k in high; intervals exclude uncertainty in the assumed linear curve shape.Source: B Exploratory comparisons along… · p. 8
  • The interpolation uses bootstrap-recomputed means, cost-selected neighbors, and weights, while Figure 2’s nominal-budget lines do not establish local linearity by realized cost.A concave strict curve would reduce the advisory gap; convexity would reverse that direction.Source: B Exploratory comparisons along… · pp. 8–9
  • At 4k, strict stopping charges unfinished terminal-prefix tokens, which cannot contribute candidates under the completed-only rule.Table 5’s fraction divides total prefix tokens by total strict cost, not by replay.Source: Table 5 · p. 9, B Exploratory comparisons along… · p. 9

C Additional accounting and selection diagnostics

Additional diagnostics show that nominal caps can produce different eligible-pool sizes and that accounting choices and selection repairs qualify the comparisons. Reconstructed prefixes sometimes contain boxes, but their correctness is not inferred; selector harms persist after grade repair in low at 32k.

C Additional accounting and selection diagnostics

  • At high 4k, advisory returns exactly one attempt in 80.50% of replays, while 768 pools are empty, 1,174 have one eligible answer, and 458 have multiple answers.Low 4k has 54 empty, 480 single-answer, and 1,866 multi-answer pools; tail costs substantially exceed mean costs.Source: C Additional accounting and selection… · p. 9, Table 6 · p. 9
  • Including prompts, low advisory 4k averages 8,100 total tokens versus 8,974 for strict 8k, while high advisory 4k averages 18,993 versus 16,510 for strict 16k and 32,695 for strict 32k.Input and output pricing, caching, serving latency, and FLOPs are not modeled.Source: C Additional accounting and selection… · p. 9
  • At 4k, reconstructed strict boundary prefixes contain any complete box in 1.78% of examined low prefixes and 6.43% of examined high prefixes, but no prefix correctness is inferred.The retokenized-text sensitivity does not prove positional alignment or reconstruct observed online cancellation.Source: C Additional accounting and selection… · p. 10, Table 7 · p. 10
  • At low 32k, maximum completion-mean log-probability fixes 14 outcomes and harms 31 of 2,400 under both source and repaired grades, with a paired mean of −0.71 pp [−1.29, −0.17].All 31 harmful events change the normalized answer.Source: C Additional accounting and selection… · p. 10
  • Changing the attempt cap has little effect in high because it seldom reaches a large cap at these budgets; ranges across 20 orders measure ordering sensitivity, not uncertainty over problems.Table 8 reports paired majority changes for nested K = 8, 16, 32 banks and canonical ascending-seed order.Source: C Additional accounting and selection… · p. 11, Table 8 · p. 11

Related papers

Loading 2609.38699v1…