PaperlayerView exact revision on arXiv

Source-linked AI summary

$R^3$-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets

Peisong Wang, Zhiwei Ma, Bowen Liu, Feixue Liu, Aochuan Chen, Chenyi Zi, Hongchuan Zeng, Yuhan Li, Jia Li

arXiv:2608.16033v1cs.CL

TL;DR

Existing benchmarks usually isolate tasks rather than testing computation allocation under a shared budget. R3-Bench compares shared-budget contest performance with matched single-problem competence and finds a persistent gap, with its oracle strictly higher in 71 of 72 cells.

  • Problem

    Existing benchmarks evaluate problems independently, leaving limited evidence about computation allocation across multiple problems sharing one budget.

  • Method

    R3-Bench evaluates six-problem contests across three domains and two settings, using matched single-problem runs to construct an equal-allocation replay and empirical oracle.

  • Results

    The oracle matches or exceeds contest scores in every cell and is strictly higher in 71, while equal allocation outperforms contest policy for three of six models under moderate tool-free pressure.

  • Takeaways & Limitations

    Single-problem competence does not guarantee effective allocation across problems, and lightweight external scheduling does not close the gap.

  • Takeaways & Limitations

    Fixed external scheduling remains far below the oracle across domains, while training an adaptive inner scheduling policy remains future work.

Abstract

from arXiv · show

In cognitive science, resource rationality asks how an agent should allocate limited computation to maximize expected value. Most reasoning and agent benchmarks use independent per-task budgets; existing shared-budget studies do not calibrate suite performance against the same model's demonstrated single-problem competence. We introduce $R^3$-Bench, which evaluates six-problem suites under shared budgets across mathematics, competitive programming, and abstract reasoning in tool-free and agentic settings. Matched single-problem response curves define an offline empirical oracle over observed successes. Across 72 main-table cells for six models, the oracle mean matches or exceeds the contest mean in all cells and is strictly higher in 71. Under moderate tool-free pressure, equal-allocation replay also exceeds contest performance for four of six models. Trajectory diagnostics reveal limited strategy updating and pressure-dependent failure patterns. In a three-model diagnostic under strong agentic pressure, at least one fixed scheduler exceeds the contest mean in six of nine cells, but no policy dominates across domains. These results expose a persistent gap between demonstrated competence and shared-budget realization.

1 Introduction

R3-BENCH evaluates whether LLMs allocate computation rationally across six-problem suites with shared budgets, rather than solving independently budgeted tasks. Its results show a persistent gap between single-problem competence and shared-budget performance, driven by limited adaptation and pressure-dependent allocation failures.

  • Motivation: Existing reasoning and agent benchmarks budget problems independently, so they do not test allocation across concurrent tasks under a shared computational constraint.Resource-rational reasoning requires deciding whether to continue, use tools, verify, switch tasks, or stop as expected gains change.
  • Benchmark design: R3-BENCH places 50 six-problem contests in each of three domains under one shared budget, evaluated in tool-free and agentic settings.Matched single-problem response curves and equal-allocation replay separate demonstrated problem-solving competence from its realization under shared budgets.
  • Results: Across 72 flagship model–setting–pressure–domain cells, the oracle matches or exceeds contest scores in every cell and is strictly higher in 71.Under moderate tool-free pressure, equal allocation also outperforms the contest policy for three of six models; neither larger budgets nor tools uniformly close the gap.
  • Behavioral diagnostics: Five of six models cover more than half the suite, but only DS-Pro exceeds the descriptive 50% reference for resource-rational strategy updates.Coverage is the share of problem slots with visible, substantive problem-specific work, while updates respond to new evidence and remaining shared budget.
  • Trajectory analysis: Under strong pressure, oracle-selected failures usually exhaust the shared budget on other problems; under moderate pressure, they more often stop after partial progress.Both failure types involve problems the model has already solved in isolation, implicating which problems receive computation and how much.

2 Related Work

Prior reasoning and agent benchmarks measure per-task competence with independent budgets, while budget-aware methods optimize resources within individual problems or across pipelines. R^3-Bench addresses this gap by evaluating shared-budget suites against matched single-problem competence across domains and settings.

  • Per-task reasoning and agent evaluation: Reasoning and agent benchmarks span multiple domains and tool-use settings but evaluate tasks independently, measuring per-task competence rather than computation allocation across a suite.Examples include mathematics, science question answering, graph problems, and competitive programming, alongside web interaction and long-horizon software execution.
  • Budget-aware reasoning and resource allocation: Budget-aware methods adapt sampling, token, or tool-use budgets within individual problems, while orchestration methods allocate resources across models or pipeline stages.Neither approach makes multiple problems compete for one shared budget.
  • Budget-aware reasoning and resource allocation: R^3-Bench compares shared-budget suite behavior with isolated-problem competence using matched response curves, equal-allocation replay, and an offline empirical oracle.The benchmark covers three domains in both tool-free and agentic settings.

3 Methodology: R3-BENCH

R3-BENCH evaluates resource-rational allocation across six-problem contests sharing a fixed computation budget. It uses matched contest and standalone evaluations across tool-free token budgets and agentic action budgets, with controlled difficulty composition and model-relative pressure.

  • Task formats: Each contest presents six problems under one shared allowance, while standalone evaluation presents problems independently to measure problem-solving ability without cross-problem allocation.In contest evaluation, models decide which problems to attempt and how much budget to allocate to each.
  • Evaluation settings: R3-BENCH evaluates the same contests in tool-free reasoning with output-token budgets and agentic interaction with action budgets, sharing pools, contests, parsing, and grading.Tool-free runs produce one free-form completion; agentic runs use commands and code in an interactive shell without correctness feedback during execution.
  • Benchmark construction: 50 contests per domain use frozen 300-problem pools and a fixed three-Easy, two-Medium, one-Hard composition, creating a controlled coverage–depth allocation trade-off.Difficulty tiers are based on average standalone output length from three reference models and are hidden from prompts.
  • Budget calibration: Model-specific calibration expresses pressure relative to each model’s unbudgeted resource use, using output tokens for tool-free reasoning and counted actions for agents.The parameter ρ represents relative pressure: ρ = 0.2 is strong pressure, whereas ρ = 0.8 is moderate pressure.
  • Outcomes and diagnostics: The primary outcome is average correct answers per six-problem contest, supplemented by problem-level accuracy and diagnostics that classify oracle-selected misses by allocation or execution cause.Miss causes include never attempting, attempting too late, stopping after partial progress, spending budget elsewhere, tool-feedback errors, formatting or finalization errors, and genuine unsolved cases.

4 Benchmarking SOTA LLMs

Across 72 model–setting–pressure–domain cells, the offline oracle matches or exceeds contest performance in every cell and is strictly higher in 71. Diagnostics attribute remaining gaps to order-sensitive spending, limited online strategy updates, pressure-dependent failure modes, and allocation quality unrelated to ECI capability distance.

  • Tool-free results: At ρ = 0.8, equal allocation outperforms contest performance for DeepSeek-V4-Pro, Qwen3.7-Max, GLM-5.2, and Claude-Opus-4.8.The oracle exceeds contest scores for all six models at both pressure levels, with a mean gap of 1.16 correct answers across 12 model–pressure pairs.
  • Main results: 72 reported cells show the offline response-curve oracle matching or exceeding contest scores in every cell and exceeding them in 71.The result reveals empirical headroom between observed per-task successes and realized shared-budget performance.
  • Tool-free diagnostics: Position 6 has lower accuracy than position 1 in all 12 series, while later positions show fewer answers and more budget truncation.Under strong pressure, the first position receives 20.5–52.9% of attributed output versus 16.9–23.5% under moderate pressure; five models concentrate spending earlier, with Qwen the exception.
  • Agentic diagnostics: In agentic trajectories, DS-Pro, Qwen, GLM, and Opus update strategies in only 38.5% to 63.4% of equal-domain macros, while Hy and GPT make none.Many trajectories observe tool feedback but continue with the existing allocation strategy, limiting online reallocation across problems.
  • Capability analysis: Capability distance carries no information about allocation quality: model pairs with similar or different ECI scores show comparable Gap Ratio spreads, and model orderings invert.In tool-free Math at ρ=0.2, the highest-ECI model has a 43.75% Gap Ratio, while a mid-ranked model has 4.32%.

5 Recovering the Allocation Gap

Lightweight online scheduling recovers part of the allocation gap but remains domain dependent: interventions beat the contest reference in six of nine model–domain cells, while no policy dominates. More complex directives and static external scheduling do not reliably improve allocation, motivating trained inner policies as future work.

  • The diagnostic uses three models spanning distinct regimes to test whether lightweight online scheduling can recover part of the allocation gap.DeepSeek-V4-Pro is a strong cross-domain solver whose Code result still has a substantial oracle gap, while GLM-5.2 is domain sensitive.
  • Six of nine model–domain cells beat the contest reference, but the contest reference, A, and B each achieve the best non-oracle score in three rows.Code improves under both interventions for every model, whereas Math and AR are model dependent and scheduling sometimes hurts.
  • B trails A in every Code row, leads in every AR row, and splits in Math, showing that verification changes coverage effects rather than uniformly improving allocation.Checks help when they resolve uncertainty but interfere when they only consume budget.
  • Hy-3’s best non-oracle scores remain far below the oracle in every domain, and its Code gain does not extend to Math or AR.Static directives redirect computation but do not replace the model’s ability to interpret progress and revise its plan.
  • Training an inner policy could condition scheduling on domain, progress quality, and remaining budget, but training and evaluation remain future work.Candidate actions include covering, continuing, switching, verifying, or stopping to reduce interference with correct model decisions.

6 Conclusion … D Evaluation Prompt Templates

R3-BENCH evaluates resource-rational reasoning across shared-budget six-problem suites, showing that single-problem competence does not guarantee effective allocation across mathematics, competitive programming, and abstract reasoning. The appendices define the datasets, protocols, parsing and judging procedures, oracle replay, and prompt templates supporting these evaluations.

  • 6 Conclusion: The benchmark places multiple problems under one shared computational budget and compares contest performance with the same model’s demonstrated single-problem competence across three domains.Resource-rational reasoning therefore requires policies for deciding when to continue, switch, and spend remaining computation.
  • 6 Conclusion: R3-BENCH finds a persistent gap between models’ response-curve oracle and shared-budget contest performance across tool-free and agentic settings.The gap reflects failures in both where and how much computation is allocated; lightweight scheduling and online adaptation help but do not close it.
  • Appendix Contents: The appendices document data sources, thinking-model protocols, answer parsing and judging, evaluation prompts, and additional shared-budget analyses.The contents also include response-curve oracle and offline knapsack replay, action accounting, budget calibration, detailed contest results, and pressure analyses.
  • E Response-Curve Oracle and Offline Knapsack Replay: The response-curve oracle and offline replay provide the benchmark’s comparison framework for reallocating observed single-problem outcomes under shared budgets.The appendix contents identify this analysis as the response-curve oracle and offline knapsack replay.
  • A Data Sources: The benchmark draws on Omni-MATH, MathNet, LiveCodeBench Pro, and Reasoning Gym to cover mathematical, competitive-programming, and procedural reasoning tasks.These sources provide Olympiad-level mathematics, multilingual and multimodal math problems, contamination-reduced programming problems, and verifiable reasoning environments.
  • C Answer Parsing and Judging: Outputs are parsed into problem-level answers and judged with domain-specific correctness procedures before aggregation into six-problem contest accuracy.Missing, malformed, unparsable, and judge or verifier failures receive zero credit, while behavioral labels remain diagnostic and do not affect scoring or oracle construction.
  • D Evaluation Prompt Templates: Evaluation prompts explicitly expose shared-budget objectives while requiring independently parseable, domain-specific final answers and allowing partial contest submission.Tool-free prompts prohibit external tools, whereas agentic prompts permit native tools but prohibit live judges, hidden tests, and reference solutions.

E Response-Curve Oracle and Offline Knapsack Replay · F Action Accounting and Tool Whitelist in the Agentic Setting · G Budget Calibration

The paper separates demonstrated single-problem competence from shared-budget allocation using offline replays, while defining counted agentic actions and calibrating domain-specific resource caps. The oracle is an empirical diagnostic based on observed response curves, not a theoretical performance bound.

  • E Response-Curve Oracle and Offline Knapsack Replay: Equal-allocation replay tests whether every problem has an observed correct attempt whose realized cost fits within one sixth of the shared contest budget.Unlike the oracle’s nominal caps, equal allocation uses realized attempt costs and excludes incorrect, malformed, unparsed, missing, or unjudged attempts.
  • E Response-Curve Oracle and Offline Knapsack Replay: Both replays are post-hoc diagnostics that reuse judged single-problem outcomes without rerunning models, generating outputs, transferring unused shares, or adapting thresholds across problems.They measure observed allocation headroom rather than guaranteeing that the same successes would recur under newly imposed caps.
  • E Response-Curve Oracle and Offline Knapsack Replay: The response-curve oracle reallocates a contest’s shared budget across six problems, selecting at most one observed grid level per problem through an exact multiple-choice knapsack.It may leave problems unfunded and can select a cheaper level when its observed success rate is higher.
  • E Response-Curve Oracle and Offline Knapsack Replay: The oracle is optimal only within the empirical grid and five-repeat replay, so its gap reflects diagnostic allocation headroom rather than a theoretical upper bound.Finite grid resolution and sampling error can omit better allocations or misestimate observed success rates.
  • F Action Accounting and Tool Whitelist in the Agentic Setting: Agentic budgets count parsed executable problem-solving actions, assigning each accepted counted command unit cost one independently of model turns or later execution success.Commands are classified as counted, free, blocked, or protocol errors; failed, timed-out, or erroring counted actions are not refunded.
  • F Action Accounting and Tool Whitelist in the Agentic Setting: The shared compute-tools policy counts interpreters, compilers, calculators, tests, solver-like programs, and data processing, while inspection, bookkeeping, file staging, and final-artifact writes remain free.The same accounting rules apply across mathematics, competitive programming, and abstract reasoning.
  • G Budget Calibration: Budget calibration uses output-token caps for pure natural-language reasoning and counted-action caps for agentic reasoning, with values taken directly from the frozen six-model cell inventory.The caps are not reconstructed from an assumed unconstrained baseline.

H Detailed Shared-Budget Contest Results

Formal shared-budget contests report difficulty-tier accuracies across standardized six-problem suites, while Gap Ratio shows little relationship with model capability as measured by ECI. No cell reaches statistical significance, and pooled explanatory power is negligible.

  • Formal contest setup: Each model–budget condition uses the same 50 six-problem suites, run five times, totaling 1500 judged runs per condition.The suites contain 150 easy, 100 medium, and 50 hard instances; E, M, H, and All report tier-specific and overall accuracies.
  • Difficulty-level accuracy breakdown: Tables 10 and 11 break down formal contest accuracy by easy, medium, and hard problem tiers, plus overall performance.The E, M, and H columns cover 750, 500, and 250 runs respectively, while All covers all 1500 runs.
  • Discriminant validity of Gap Ratio: No cell reaches p < 0.05; mean Spearman correlation is −0.29 (p = 0.37), and pooled within-cell R2 is 0.01 (p = 0.70).These results support the conclusion that capability ordering explains little of the allocation-gap ordering.

I Model Interfaces and Experimental Configuration … J.3 Wasted Output

The appendix specifies tool-free and agentic evaluation interfaces and analyzes budget-pressure behavior through oracle portfolios, selected misses, output allocation, and wasted reasoning. Across diagnostics, oracle selections favor easier problems, early positions absorb substantial output, and 41.2–74.6% of attributed output is spent on ultimately incorrect problems.

  • I Model Interfaces and Experimental Configuration: Tool-free evaluation returns final answers without execution or feedback, whereas agentic evaluation uses the Harbor/Terminus-2 shell under Appendix F action-accounting rules.
  • I Model Interfaces and Experimental Configuration: Models use provider-recommended sampling configurations without R3-BENCH tuning, with five independent runs per problem and thinking disabled for GPT-5.5 and Claude-Opus-4.8.
  • J Additional Analyses under Budget Pressure: Analyses compute statistics within domains and then average them equally across eligible domains using an available-domain macro.
  • J.1 Oracle Portfolio: Easy problems exceed their 50% suite share in all 12 curve-oracle portfolios, while Hard problems remain below their 1/6 share in every portfolio.
  • J.2 Selected Misses and Output Accrual by Position: At least one of the Easy or Medium selected-miss rates exceeds the Hard rate in all 12 model–budget panels, and both do so in 11.
  • J.2 Selected Misses and Output Accrual by Position: Under strong pressure, the first position consumes 20.5–52.9% of attributed output, versus 16.9–23.5% under moderate pressure; Qwen is the sole exception to the strong-pressure increase.
  • J.3 Wasted Output: Total wasted output ranges from 41.2% to 74.6% across 12 available-domain model–budget panels and is lower at ρ = .8 than at ρ = .2 for every model.

J.4 Omission and Position Diagnostics

Omission diagnostics show that pressure shifts failures toward budget truncation, while later suite positions increasingly go unanswered and become truncated. Capability ordering explains little of the allocation-gap ordering, indicating that competence alone poorly predicts shared-budget allocation.

  • Omission diagnostics: Under moderate pressure, all six models answer more slots and truncate fewer, whereas strong-pressure truncation spans 8.7%–68.0% across model macros.Under moderate pressure, truncation ranges from 2.5% to 19.3%.
  • Position diagnostics: From position 1 to position 6, answered rates fall and judged budget-truncation rates rise in all 12 model–pressure series, usually more sharply under strong pressure.Figure 12 provides the trajectory-level counterpart to the accuracy curves.
  • Discriminant validity: Mean capability ordering explains little of allocation-gap ordering: global Spearman correlation is −0.29 (p = 0.37), and pooled within-cell R2 is 0.01 (p = 0.70).The discriminant-validity analysis uses five models and 120 permutations.

K Human Annotation of Trajectories

Human annotators exhaustively labeled 18,000 tool-free and agentic trajectories across 36 model–pressure–domain cells using evidence-based behavioral rubrics. The annotations distinguish oracle-gap causes, adaptation, engineering issues, and decision-regret patterns without claiming causal effects.

  • Scope and annotation unit: 18,000 trajectories across 36 cells were exhaustively annotated by ten human annotators using a shared codebook and standardized trajectory records.Each cell contained 50 contests with five independent trajectories in both tool-free and agentic settings; annotators were assigned 1,800 trajectories each.
  • Annotation procedure: Every positive behavioral label required an exact supporting quotation and step or event identifier, while absent or speculative evidence mandated negative or insufficient-evidence labels.Verifier correctness, reward, and oracle fields were frozen inputs that annotators could not revise.
  • Oracle-gap attribution rubric: Oracle-selected misses received one primary attribution—such as never attempted, attempted too late, partial progress, budget spent elsewhere, tool-feedback errors, formatting failures, or genuinely unsolved—with insufficient cases excluded.Reported shares therefore use behavior-attributable misses rather than all misses.
  • Online-adaptation rubric: Online-adaptation labels required explicit difficulty or feedback observation, a later substantive strategy revision, and, for resource-rational updates, an explicit causal link to feedback, value, cost, probability, difficulty, or opportunity cost.Attempted-but-failed updates required the revised strategy to be enacted and the same shortfall to remain visible later.
  • Decision-regret rubric: Decision-regret labels captured undercoverage, overcommitment, late budget exhaustion, premature shelving, and weak finalization from concrete trajectory evidence, but characterized visible patterns rather than causal effects.Each pattern required specific evidence, such as feasible uncovered work, disproportionate spending despite weak feedback, avoidable exhaustion, or submission-ready but missing or malformed output.

L Behavioral Coordinate … N.2 Case II: More Attributed Output than a Successful Standalone Run

The merged sections diagnose shared-budget failures through behavioral coordinates, scheduler interventions, and trajectory case studies. They show that broad problem coverage does not ensure adaptive allocation, while simple guards can target overcommitment and individual failures can persist despite more attributed output than a successful standalone run.

  • L Behavioral Coordinate: Broad coverage does not by itself imply resource-rational reasoning: models may work on many problems while rarely revising budget allocation from new evidence.Figure 1 combines tool-free and agentic settings, both pressures, and all available domains; its quadrant labels describe coordinate regions rather than latent model types.
  • M Online Scheduler Directives: The scheduler study compares the contest reference with a coverage guard and a coverage-plus-verification guard while holding the model, suite, budget, tools, parser, and judging protocol fixed.The coverage guard requires every visible problem to receive an initial paid probe before repeated investment, while the verification gate limits repeated checking after a stable candidate unless it changes.
  • N Trajectory-Level Case Studies of Allocation Failure and Recovery: The trajectory case studies illustrate aggregate allocation-failure mechanisms using 14 manually audited candidate cases selected for formal outcomes, complete trajectories and artifacts, and no detected infrastructure failure.Final correctness comes from domain-specific parsers or verifiers; behavioral labels diagnose trajectories, while oracle evidence indicates observed allocation headroom rather than guaranteed reproducibility in one contest trajectory.
  • N.1 Case I: Local Debugging Crowds Out Four Oracle-Selected Problems: DeepSeek-Chat spends all ten counted actions on Problem C, scores 0/6, and leaves five problems untouched, whereas the cap-limited oracle reaches 4/6 by selecting four other problems.Problems A, B, D, and E each have an accepted observed response-curve point at two actions, yielding a unique eight-action selected optimum and four displaced empirically reachable opportunities; Problem C is from an easy benchmark tier.
  • N.2 Case II: More Attributed Output than a Successful Standalone Run: In Code Set 9 at ρ = 0.8, DeepSeek-Chat solves two contest problems although the response-curve oracle selects three, including Problem F, which it solves standalone in 173 output tokens.The contest attributes 322.4 Stage-1 tokens to Problem F, yet its complete program is wrong; this is a section-length attribution, while 173 tokens is the lowest sampled successful point.
  • N.2 Case II: More Attributed Output than a Successful Standalone Run: The contest failure is a task misinterpretation rather than truncation or incomplete implementation: Stage 1 ends after 1,618 of 5,065 available tokens, but the model incorrectly requires every character after the initial 10 to be 0.The standalone solution accepts a nonzero exponent suffix, whereas the contest program implements the stronger all-zero rule and receives a wrong-answer verdict; Stage 2 only extracts the same incorrect code.

N.3 Case III: Protocol-Shaped Shelving Followed by Weak Finalization · N.4 Matched Intervention Example: Broader Coverage with Two Additional Accepted Answers

A matched intervention shows that broader coverage can add two accepted answers under the same seven-action budget, while an abstract-reasoning case shows weak finalization after a known defect is identified. Together, the cases illustrate both the benefits and costs of coverage-first scheduling and the limits of interpreting the shelving failure as an oracle-recoverable gap.

  • N.3 Case III: Protocol-Shaped Shelving Followed by Weak Finalization: The initial switch away from Problem 3 is protocol-shaped because other problems lacked initial probes, whereas the later refusal to return reflects weak finalization rather than an oracle-recoverable gap.After coverage is complete, the guard no longer blocks returning to the known defect; no linked per-contest response-curve oracle record exists.
  • N.3 Case III: Protocol-Shaped Shelving Followed by Weak Finalization: At rho = 0.8, DS-Pro uses six of seven actions, solves three of six abstract-reasoning problems, and terminates with one action unused.This run is outside the fixed rho = 0.2 scheduler-intervention cells.
  • N.3 Case III: Protocol-Shaped Shelving Followed by Weak Finalization: The agent identifies that its palindrome candidate omits letters, but after all problems are covered it keeps the invalid answer despite one paid action remaining.The candidate has 65 letters versus 68 in the input and omits one occurrence each of h, k, and r; the post-hoc check was not exposed during the run.
  • N.4 Matched Intervention Example: Broader Coverage with Two Additional Accepted Answers: 2/6 to 4/6: Strategy B outperforms the contest reference on the matched coding contest under the same seven-action budget.Both strategies execute seven counted actions; the contest reference scores 2/6, while Strategy B scores 4/6.
  • N.4 Matched Intervention Example: Broader Coverage with Two Additional Accepted Answers: The matched comparison uses DS-Pro on Code Set 12 with the same seven-action budget, tools, parser, verifier, and reported A0.2 configuration.Strategy B adds the Appendix M coverage guard and verification gate, while the contest reference is the baseline agent.
  • N.4 Matched Intervention Example: Broader Coverage with Two Additional Accepted Answers: Strategy B accepts Problems D and E, which the contest reference never touches, by reallocating actions from concentrated early depth to broader coverage.The contest allocates (3, 3, 0, 0, 0, 1), whereas Strategy B allocates (2, 1, 1, 1, 1, 1).
  • N.4 Matched Intervention Example: Broader Coverage with Two Additional Accepted Answers: The broader-coverage comparison is illustrative rather than deterministic because the trajectories are independent model samples, and its mechanism does not transfer uniformly across domains.Blocked reinvestments consume no budget, but the coverage guard still requires a paid probe on each problem.
  • N.4 Matched Intervention Example: Broader Coverage with Two Additional Accepted Answers: Coverage consumes six of seven actions, leaving Strategy B one action for additional depth on Problem A while costing depth on Problems B and C.Problem B retains its accepted answer with two fewer actions, while one action is spent on unsolved hard Problem C.

O Target-in-Suite Context-Stress Diagnostic

The target-in-suite diagnostic tests whether presenting six problems causes context interference independent of cross-problem resource allocation. Across 12 cells, suite context generally had little effect on target-only performance, though Qwen3.7-Max showed larger Coding differences.

  • Experimental design: The diagnostic preserves the six-problem suite context but removes the need to select among or allocate resources across multiple actionable problems.Each target position is evaluated once, pairing stress outcomes with problem-matched single-problem references.
  • Experimental design: The experiment evaluates Qwen3.7-Max and DeepSeek-V4-Pro in tool-free and agentic mathematics, competitive programming, and abstract-reasoning settings.It uses 10 deterministically sampled suites per domain, shared across models and settings, yielding 720 stress episodes.
  • Controls and analysis: The authors treat the analysis as a robustness diagnostic rather than a causal decomposition of the contest–oracle gap.Control–stress differences are averaged within suites and then equally across the 10 suites.
  • Results: 1.1 percentage points was the unweighted mean control–stress difference across 12 model–setting–domain cells, with nine differences at most 5 points.All six DeepSeek-V4-Pro cells differed by at most 5 points, while Qwen3.7-Max had larger differences in tool-free Coding (6.7 points) and Agentic Coding (8.3 points).
Loading 2608.16033v1…