$R^3$-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets

1 Introduction

• Five of six models cover more than half the suite, but only DS-Pro exceeds the descriptive 50% reference for resource-rational strategy updates.Broad problem coverage does not imply resource-rational behavior.Source: Figure 1 · p. 1, §1 · p. 2
• Existing reasoning and agent benchmarks allocate resources per problem or task, so they do not test the opportunity cost of spending computation on one task rather than another.With shared budgets, an agent must weigh further thought, tool calls, tests, verification, task switching, or stopping against the resources left for other tasks.Source: §1 · pp. 1–2
• R3-BENCH evaluates 50 six-problem contests in mathematics, competitive programming, and abstract reasoning, in tool-free and agentic settings under shared budgets.Matched single-problem runs support equal-allocation replay and an offline empirical response-curve oracle that selects demonstrated successes subject to the contest budget.Source: §1 · p. 2
• The oracle matches or exceeds contest scores in all 72 flagship cells and is strictly higher in 71; equal allocation beats contest performance for three of six models under moderate tool-free pressure.Neither larger budgets nor tool access uniformly closes the gap.Source: §1 · p. 2
• Under strong pressure, oracle-selected problems missed in contests most often fail because other problems exhaust the shared budget; under moderate pressure, they more often fail after partial progress.These failures concern problems already solved in isolation, implicating allocation rather than a capability limit.Source: §1 · p. 2
• Fixed online schedulers achieve higher mean performance in six of nine model–domain cells, but no policy transfers uniformly across domains.Models within two points on a composite of over fifty benchmarks differ by up to 60.6 percentage points in Gap Ratio.Source: §1 · p. 2

Per-task reasoning and agent evaluation

• Reasoning and agent benchmarks measure per-task competence under independent budgets, despite spanning multiple domains and tool-using, long-horizon tasks.Reasoning benchmarks include mathematics, science question answering, graph problems, and competitive programming; agent benchmarks add tool use, web interaction, and long-horizon software execution.Source: §2 · pp. 2–3

Budget-aware reasoning and resource allocation

• Budget-aware methods adapt sampling, token, or tool-use budgets within individual problems, while orchestration allocates resources across models or pipeline stages.Neither approach makes multiple problems compete for one shared budget.Source: §2 · p. 3
• USACOArena studies credit-budgeted coding agents, while CLEAR (Wan et al., 2026) allocates tokens across queries; neither measures allocation against per-task reasoning ability.Table 1 compares representative evaluations by setting, domain, and shared-budget coverage; its checkmarks indicate protocol coverage, not methodological superiority.Source: §2 · p. 3, Table 1 · p. 3
• R3-Bench combines shared-budget suites with matched single-problem response curves, equal-allocation replay, and an offline empirical oracle across three domains and tool-free and agentic settings.Its motivating question is how much of a model’s isolated-problem competence its own shared-budget behavior realizes.Source: §2 · p. 3

3 Methodology: R3-BENCH

• Each six-problem contest shares one resource budget, making spending on one problem reduce what remains for the others.Models must choose which problems to attempt, how much computation to invest, and when to continue, switch, or stop.Source: §3 · p. 3

3.1 Task-Suite Construction

• Each domain uses 300 problems stratified by output length into tiers, then 50 contests with three Easy, two Medium, and one Hard problem.Three reference models’ average standalone output lengths determine the fixed tiers, which are hidden from prompts; the same problems and contests support per-task and contest evaluation.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 Tool-free reasoning Setting

• Tool-free reasoning uses a single free-form completion without tools, shell access, code execution, or interactive feedback, with budgets measured in output tokens.The two evaluation settings share problem pools, difficulty tiers, contest construction, answer parser, and grading protocol, differing in runtime and budget unit.Source: §3 · p. 3, §3.2 · p. 5

3.3 Agentic Setting

• Agentic evaluation uses tool-using models in an interactive shell, measures resources in counted tool actions, and determines correctness only after submission.Agents can execute commands and code, inspect intermediate results, and submit final answers through designated artifacts; no official correctness feedback is available during the run.Source: §3.3 · p. 5

3.4 Metrics

• Contest score is the average correct answers per six-problem contest; allocation quality is assessed against equal-allocation replay and an offline response-curve oracle.The oracle selects a budget level, including zero, for each problem to maximize expected correct answers under the total budget, using isolated runs at multiple budget levels.Source: §3.4 · pp. 5–6
• Diagnostics classify oracle-selected misses by primary cause and annotate whether agents observe relevant evidence, update strategy, and respond resource-rationally.Miss causes include not attempting a problem, attempting too late, partial progress, spending budget elsewhere, misreading feedback, incomplete finalization, or being genuinely unsolved; insufficient-evidence cases are excluded.Source: §3.4 · p. 6

4 Benchmarking SOTA LLMs

• The benchmark covers eight frontier LLMs across two settings, three domains, and two budget pressures; Table 2 reports six flagship models.Complete eight-model results are in Appendix H.Source: §4 · p. 6

4.1 Overall Results

• In a target-in-suite diagnostic, nine of 12 cells had point-estimated losses of at most 5 percentage points, with a 1.1-point mean control–stress difference.The diagnostic used two models across both settings and all three domains, showing all six problems while designating only one for solution.Source: §4.1 · p. 6

4.2 Tool-free Reasoning

• In tool-free runs, the oracle exceeded contest scores for all six models at both pressures, with a mean gap of 1.16 correct answers across 12 model–pressure pairs.The oracle reuses observed per-task successes under the same total budget, quantifying offline empirical headroom rather than executable-policy performance.Source: §4.2 · p. 6, Figure 3 · p. 8
• At ρ = 0.8, equal allocation outperformed the contest for four of six models, but under strong pressure its per-problem shares could become extremely small.For GPT-5.5 at ρ = 0.2, the replay thresholds were 38, 47, and 6 output tokens for Math, Code, and AR, respectively, leaving its equal-allocation score near zero.Source: §4.2 · pp. 6–7, Figure 3 · p. 8

4.1 Overall Results

• Across 72 model–setting–pressure–domain cells, the offline oracle matched or exceeded contest scores in every cell and was strictly higher in 71.The comparison is reported in Table 2; it shows empirical headroom from observed per-task successes, not an executable policy’s performance.Source: §4.1 · p. 6, Table 2 · p. 7

4.2 Tool-free Reasoning

• Position 6 had lower accuracy than position 1 in all 12 series, consistent with spending following arrival order rather than problem worth.Under strong pressure, position 1 received 20.5–52.9% of attributed output versus 16.9–23.5% under moderate pressure; five models showed stronger early concentration at ρ = 0.2, with Qwen the exception.Source: §4.2 · p. 7, Figure 4 · p. 8

4.3 Agentic Reasoning

• Substantive strategy updates occurred in only 38.5%–63.4% of equal-domain macro trajectories for DS-Pro, Qwen, GLM, and Opus; Hy and GPT made none.Many trajectories observed tool feedback but continued with the existing strategy rather than reallocating budget across problems.Source: §4.3 · pp. 7–8
• Under ρ = 0.2, spent-budget-elsewhere dominated four models; under ρ = 0.8, stopped-after-partial-progress dominated four others, showing that misallocation persisted but shifted.At strong pressure, spent-budget-elsewhere accounted for 42% of GLM, 95% of Hy, 66% of GPT, and 59% of Opus misses. At moderate pressure, stopped-after-partial-progress accounted for 67% of DS-Pro, 38% of GLM, 38% of Hy, and 55% of GPT misses; these stops indicate under-investment, not capability limits.Source: §4.3 · p. 8

• Pairs within 3 ECI points differ by 9.6–21.0 percentage points in mean absolute Gap Ratio, versus 12.0–19.2 points for pairs more than 4 points apart.Orderings also invert: in tool-free Math at ρ=0.2, the highest-ECI model has the largest Gap Ratio (43.75%), while a mid-ranked model has the smallest (4.32%; Table 12 in Appendix H).Source: GLM Hy · p. 9

5 Recovering the Allocation Gap

• Under strong pressure (ρ = 0.2), Strategy A requires a focused probe of every visible problem before a second paid step, while Strategy B adds a verification gate for stable candidates.Neither intervention receives answer labels, difficulty labels, response curves, or oracle-selected problems; both use only runtime-visible coverage, budget, and candidate-state information.Source: §5 · p. 9
• 6 of 9 model–domain cells beat the contest reference under an intervention, but no policy dominates: the contest reference, A, and B each have the best non-oracle score in 3 rows.Code improves under both interventions for every model, whereas Math and abstract reasoning (AR) are model dependent and scheduling sometimes hurts.Source: §5 · p. 9
• B trails A in every Code row, leads in every AR row, and splits in Math, showing that verification changes coverage’s effect rather than uniformly improving allocation.The gate helps when checks resolve uncertainty and interferes when they consume budget without doing so.Source: §5 · p. 10
• Hy-3’s best non-oracle scores remain far below the oracle in all domains, and its Code gain does not extend to Math or AR.Static directives can redirect computation but do not replace the model’s ability to interpret progress and revise its plan.Source: §5 · p. 10
• The intervention study motivates a trained inner policy that conditions scheduling on domain, progress quality, and remaining budget, but training and evaluation remain future work.The proposed policy would choose when to cover, continue, switch, verify, or stop; the study highlights the limited cross-domain generality of a single fixed directive.Source: §5 · p. 10

6 Conclusion

• Across tool-free and agentic settings, current models leave a persistent gap to their response-curve oracle, so demonstrated single-problem competence does not ensure effective allocation across a suite.The benchmark covers mathematics, competitive programming, and abstract reasoning, comparing contest performance with what the same model has demonstrated on single problems.Source: §6 · p. 10
• Under strong pressure models often spend budget on other problems, while under moderate pressure they more often stop after partial progress; scheduling can help but does not close the gap.Broader coverage helps when runtime feedback is informative but can hurt when shallow probes provide weak evidence; the conclusion calls for policies that decide when to continue, switch, and allocate remaining budget.Source: §6 · pp. 10–11

A Data Sources

• The data sources span Olympiad mathematics, competitive programming, and procedural reasoning environments, supporting evaluation across problem topics, difficulty, and reasoning domains.Omni-MATH has 4,428 problems across more than 33 subdomains; MathNet has 30,676 problems across 47 countries and 16 languages; LiveCodeBench Pro has 584 problems; Reasoning Gym offers over 100 generators.Source: A Data Sources · p. 13

B Protocol for Thinking Models

• Stage 1 is the problem-solving stage, using a single-problem budget or one shared budget across six contest problems; across 300 audited Stage 2 calls, none re-solved a problem or introduced new derivations or algorithms.Stage 2 converts the stopped Stage 1 output into the required answer format; its output is logged separately rather than charged, and every audited output was attributable to Stage 1 material.Source: B Protocol for Thinking Models · pp. 13–14

C Answer Parsing and Judging

• Domain-specific parsers and judges assign problem-level correctness, aggregate contest accuracy, and treat parser or verifier failures as zero credit; behavior labels are diagnostic only.Numeric domain scores are binarized as full credit for the main accuracy tables, and the response-curve oracle reuses judged outcomes rather than introducing a new judge.Source: C Answer Parsing and Judging · pp. 14–16

D Evaluation Prompt Templates

• Evaluation prompts layer reusable instructions before run-specific content to enable prefix-cache reuse and do not expose oracle decisions, reference answers, hidden-test outcomes, or judge feedback.Tool-free templates use domain-specific answer formats, while contest prompts present all six problems and a shared output-token budget.Source: D Evaluation Prompt Templates · p. 16, §D.1 · p. 17

D.1 Tool-free Reasoning

• Tool-free contest prompts ask models to maximize correct answers within a shared budget, choose problems and effort allocation, and return parseable outputs under domain-specific contracts.The prompts prohibit external tools and allow partial submissions; unsubmitted problems receive zero credit.Source: §D.1 · p. 17, Table 6 · p. 17
• Tool-free prompts allocate one reasoning-token budget across the contest, allow submissions for any problem subset, and require domain-specific answer formats.Mathematics uses \boxed{...}, while abstract reasoning uses <answer>...</answer>; problems without valid answers receive zero credit.Source: §D.1 · pp. 17–18

D.2 Reasoning-Model Two-Stage Protocol

• Thinking-model runs separate budgeted reasoning from answer formatting: Stage 1 records candidate answers or No answer, and Stage 2 formats only from the stopped scratchpad.For coding, the finalizer returns MISSING if the trace cannot support a complete standalone program.Source: §D.2 · p. 18

D.3 Agentic Prompts

• Agentic prompts give native-tool agents a shared counted-action budget, permit partial submissions, and require a contest-level answer artifact and completion action.Only designated counted actions consume the budget; the task permits local computation and tests but prohibits access to live-judge or hidden evaluation resources.Source: §D.3 · pp. 18–19

[FINALIZATION PROMPT]

• The finalization prompt preserves valid submissions, prohibits new problem-solving, and requires valid formatted answers and the contest-level artifact before completion.Source: [FINALIZATION PROMPT] · p. 19

D.4 Judging Prompts

• Mathematics answers are checked by an offline equivalence judge, abstract reasoning by the Reasoning Gym extractor and scorer, and coding by executable tests.The mathematics judge is DeepSeek V4 Flash (DeepSeek-AI et al., 2026), and its prompt is not shown to solvers.Source: §D.4 · pp. 19–20

D.5 Scheduler Prompt Variants

• The scheduler comparison uses a baseline without a scheduler guard, Strategy A with coverage, and Strategy B with coverage plus a verification gate.The shared template specifies six visible problems, one initial probe per problem, and at most one additional check of an unchanged candidate; the comparison excludes oracle scheduling and stop-finalization scheme D.Source: §D.5 · p. 20

E Response-Curve Oracle and Offline Knapsack Replay

• The equal-allocation replay counts problems with an observed correct attempt costing no more than one sixth of the contest budget, using realized attempt costs.Neither replay reruns the model or generates answers; the oracle uses nominal grid caps, whereas equal allocation uses realized costs.Source: E Response-Curve Oracle and Offline… · pp. 20–21
• Both replays are post-hoc diagnostics, not guarantees of performance under newly imposed caps or a theoretical upper bound on contest performance.They rely on collected attempts, and the oracle is limited to its finite grid and five repeats per level.Source: E Response-Curve Oracle and Offline… · pp. 21–22
• The response-curve oracle reallocates the shared budget across six problems, selecting at most one grid level per problem to maximize observed success rates.It solves a multiple-choice knapsack exactly by enumerating all 46,656 assignments, without assuming that success rates rise with budget level.Source: E Response-Curve Oracle and Offline… · pp. 21–22

F Action Accounting and Tool Whitelist in the Agentic Setting

• In agentic runs, each accepted problem-solving command costs one budget unit, even if it errors or times out; non-compute bookkeeping and file staging are free.The shared compute-tools policy counts operations such as interpreters, compilers, tests, and data processing, regardless of how many such operations one command contains.Source: F Action Accounting and Tool Whitelist… · p. 23
• After compute-budget exhaustion, further compute commands are blocked, but agents may still finalize answers; only designated final artifacts receive credit.Correctness feedback is unavailable during runs, and live-judge or submit-style commands are blocked.Source: F Action Accounting and Tool Whitelist… · p. 23

G Budget Calibration

• Budget Calibration reports output-token caps for pure natural-language runs and counted-action caps for agentic runs, taken from the frozen six-model cell inventory.The formal shared-budget caps are listed in Table 8.Source: G Budget Calibration · pp. 24–25, Table 8 · p. 25

Difficulty-level contest accuracy

• Tables 10 and 11 report difficulty-tier accuracy across 50 six-problem suites per model–budget condition, each run five times.The suites contain 150 easy, 100 medium, and 50 hard instances; E, M, H, and All accuracy use 750, 500, 250, and 1,500 judged runs, respectively.Source: H Detailed Shared-Budget Contest Results · p. 25

Discriminant validity of Gap Ratio

• No cell reaches p < 0.05; mean Spearman correlation is −0.29 (p = 0.37), and pooled within-cell R^2 is 0.01 (p = 0.70).These results support the construct-validity analysis of the R3-BENCH Gap Ratio.Source: H Detailed Shared-Budget Contest Results · p. 25

I Model Interfaces and Experimental Configuration

• The benchmark uses text-only tool-free responses or agentic runs in a Harbor/Terminus-2 shell environment; only the latter provide shell access and interactive feedback.Action accounting for agentic runs is specified in Appendix F.Source: I Model Interfaces and Experimental… · p. 25
• Each model has five independent runs per problem or contest instance, with five complete trajectories for agentic evaluation; sampling settings follow provider recommendations and are not tuned on R3-BENCH.Thinking is disabled for GPT-5.5 and Claude-Opus-4.8 because their APIs do not expose the corresponding chain-of-thought content in the evaluation interfaces.Source: I Model Interfaces and Experimental… · p. 25

J Additional Analyses under Budget Pressure

• Each additional-analysis statistic is computed within a domain, then averaged equally over domains eligible for that metric; this is called an available-domain macro.Source: J Additional Analyses under Budget… · p. 25

J.2 Selected Misses and Output Accrual by Position

• In all 12 model–budget panels, at least one of the Easy or Medium selected-miss rates exceeds the Hard rate; both rates do so in 11 panels.Selected misses are problems chosen by the curve oracle but missed in the contest; rates divide their count by all judged problems in the paired suites.Source: §J.2 · p. 26
• 20.5–52.9% of attributed output goes to the first position under strong pressure, versus 16.9–23.5% under moderate pressure; Qwen is the only model without a larger strong-pressure share.Attributed output is the budget-charged output; for thinking models, it is Stage 1 completion tokens, including reasoning tokens.Source: §J.2 · p. 26

J.3 Wasted Output

• 41.2%–74.6% of attributed output is spent on ultimately incorrect problems, and total waste is lower at ρ = .8 than at ρ = .2 for all six models.The difficulty composition of this waste varies across models and budgets.Source: §J.3 · pp. 26–27

J.1 Oracle Portfolio

• Easy problems exceed their 50% suite share in all 12 curve-oracle portfolios, while Hard problems remain below their 1/6 share in all 12.Table 9 reports average correct answers per six-problem suite for Contest and Oracle, with Gap Ratio defined as (Oracle − Contest)/Oracle; lower is better.Source: §J.1 · p. 25, Table 9 · p. 26, Figure 7 · p. 27

J.4 Omission and Position Diagnostics

• Under moderate pressure, all six models have higher answered shares and lower truncation shares; truncation ranges from 2.5% to 19.3%, versus 8.7% to 68.0% under strong pressure.Figure 11 decomposes problem slots into answered/not missing, truncation, formatting or alignment failure, unresolved attempts, and residual evidence-insufficient cases.Source: §J.4 · pp. 27–28, Figure 11 · p. 31
• Across all 12 model–pressure series, answered rates fall and budget-truncation rates rise from position 1 to position 6, usually more under strong pressure.Figure 12 reports the position trends; the prose notes the stronger-pressure pattern is usually larger.Source: §J.4 · p. 28, Figure 12 · p. 32

K Human Annotation of Trajectories

• Ten annotators exhaustively labeled 18,000 trajectories spanning six models, two pressures, three domains, and tool-free and agentic settings.The corpus covers 36 model–pressure–domain cells; trajectories were randomly divided evenly among annotators, with model identity and difficulty tiers withheld.Source: K Human Annotation of Trajectories · p. 29
• Annotation used standardized trajectory records and explicit evidence checks, while correctness, reward, and oracle metadata remained frozen and annotators did not re-solve problems.Every positive behavioral label required an exact supporting quotation and step or event identifier; absent or speculative evidence required a negative or insufficient-evidence label.Source: K Human Annotation of Trajectories · p. 29
• Each oracle-selected miss received one primary attribution, while misses lacking sufficient metadata or behavioral evidence were excluded from behavior-attributable shares.Attribution categories include never attempted, attempted too late, stopped after partial progress, spent budget elsewhere, misread tool feedback, wrong format or incomplete finalization, and genuinely unsolved.Source: K Human Annotation of Trajectories · p. 30
• Online-adaptation labels distinguish difficulty observation, substantive strategy updates, resource-rational updates, and attempted-but-failed updates.A substantive update requires an enacted strategy, a new observation, and a subsequent substantive revision; a resource-rational update additionally links revision to feedback, expected value, cost, success probability, difficulty, or shared-budget opportunity cost.Source: K Human Annotation of Trajectories · p. 31
• Five descriptive regret patterns capture undercoverage, overcommitment, late budget exhaustion, premature shelving, and weak finalization.These require trajectory evidence of uncovered feasible work, disproportionate spending despite weak feedback, avoidable exhaustion blocking intended work, shallow abandonment despite a feasible next action, or missing or malformed submission despite submission-ready content.Source: K Human Annotation of Trajectories · p. 32
• The behavioral labels characterize evidence visible in trajectories and do not identify causal effects.Source: K Human Annotation of Trajectories · p. 32

L Behavioral Coordinate

• Broad coverage does not imply resource-rational reasoning: Figure 1 separates confirmed problem coverage from the unconditional confirmed strategy-update rate across settings, pressures, and domains.Its four named quadrants are descriptive coordinate regions, not latent model types.Source: L Behavioral Coordinate · pp. 32–33

M Online Scheduler Directives

• Strategies A and B add runtime-visible scheduling constraints: both require an initial probe of every visible problem, while B also limits repeated checking of a stable candidate.The directives use no answer labels, hidden correctness feedback, response-curve information, oracle-selected problems, or difficulty labels.Source: M Online Scheduler Directives · pp. 33–34
• The scheduler study tested DeepSeek-V4-Pro, GLM-5.2, and Hy-3 at strong pressure ρ = 0.2, using 50 six-problem suites per model, domain, and configuration.Scores are mean correct answers per six-problem suite; missing or unscored suites count as zero.Source: M Online Scheduler Directives · p. 34

N Trajectory-Level Case Studies of Allocation Failure and Recovery

• The trajectory cases were selected from 14 shortlisted candidates for complete evidence and mechanistic distinctness; oracle evidence indicates observed allocation headroom, not guaranteed reproducibility in one contest trajectory.Final correctness comes from domain-specific parsers or verifiers, while behavioral labels are used only for retrospective diagnosis.Source: N Trajectory-Level Case Studies of… · p. 34

N.1 Case I: Local Debugging Crowds Out Four Oracle-Selected Problems

• The contest scored 0/6 after spending all ten actions on Problem C, while the response-curve oracle selected four other problems for 4/6 at eight observed actions.Problem C was in an easy benchmark tier; the agent responded to local errors but did not reassess whether another problem was a better use of the remaining budget.Source: §N.1 · pp. 34–35, Table 15 · p. 35

N.2 Case II: More Attributed Output than a Successful Standalone Run

• 322.4 Stage-1 attributed tokens versus 173 standalone tokens accompanied a wrong contest program: it imposed a stricter suffix rule, although the standalone program was accepted.Stage 1 ended normally, so token-cap truncation did not cause the failure; 173 is the lowest sampled successful point, and 322.4 is a section-length attribution rather than an API counter.Source: §N.2 · p. 35

N.3 Case III: Protocol-Shaped Shelving Followed by Weak Finalization

• DS-Pro’s shelved palindrome used 65 letters for a 68-letter input, omitting one each of h, k, and r; the agent had noted the defect.A deterministic post-hoc multiset check confirmed the mismatch, but the check was not exposed to the model during the run.Source: §N.3 · p. 36
• After all six problems were covered, DS-Pro kept a known-invalid palindrome answer despite one remaining action; it scored 0.05 and counted as incorrect.The initial reinvestment attempt was blocked by the coverage guard, but the later choice to stop investigating was model-chosen; no linked response-curve oracle record supports a numerical recoverable-gap claim.Source: §N.3 · p. 36

N.4 Matched Intervention Example: Broader Coverage with Two Additional Accepted Answers

• Strategy B scores 4/6 versus the contest reference’s 2/6 on the same coding contest, with Problems D and E accepted only under Strategy B.The matched comparison uses the same model, Code Set 12, and seven counted actions.Source: Table 17 · p. 36, §N.4 · p. 36
• The contest reference used allocation (3, 3, 0, 0, 0, 1), while Strategy B used (2, 1, 1, 1, 1, 1) across Problems A–F.Strategy B’s coverage guard blocked three attempted reinvestments before execution, and those commands consumed no budget.Source: §N.4 · p. 36
• Covering all six problems consumed six of seven actions, leaving one for depth; Strategy B spent it on Problem A, while Problem B retained acceptance with two fewer actions.One action went to unsolved hard Problem C, illustrating the cost of breadth; the trajectories were independent model samples, not a deterministic counterfactual.Source: §N.4 · pp. 36–37
• Coverage directives improve coding results for every model, but help some models and hurt others in mathematics and abstract reasoning; the observed change does not isolate the verification gate.The authors attribute the example’s mechanism primarily to the coverage constraint.Source: §N.4 · p. 37

Motivation and design

• The target-in-suite diagnostic retained all six problem statements while removing the need to select among or allocate resources across multiple actionable problems.Only the designated target was scored; the other five statements were marked as distractors.Source: O Target-in-Suite Context-Stress… · p. 37

Experimental design

• The diagnostic tested Qwen3.7-Max and DeepSeek-V4-Pro across three domains and two settings using 10 shared suites and 60 target episodes per model–setting–domain cell.Each of six positions served once as the target; the model saw all six statements in their original order, while the other five were distractors. Agentic distractor computation was blocked, and each target received the largest frozen single-problem budget.Source: O Target-in-Suite Context-Stress… · p. 37

Controls and analysis

• Stress outcomes were paired with problem-matched single-problem references, averaged across six pairs within each suite, and then averaged equally over 10 suites.The authors frame this as a robustness diagnostic, not a causal decomposition of the contest–oracle gap.Source: O Target-in-Suite Context-Stress… · p. 37

Results

• Across 12 cells, nine had control–stress differences of at most 5 percentage points (mean 1.1 percentage points), while Qwen3.7-Max Coding differences were 6.7 points tool-free and 8.3 points agentic.All six DeepSeek-V4-Pro cells had differences of at most 5 points.Source: Table 18 · p. 37, O Target-in-Suite Context-Stress… · p. 37