Where Should a Stuck Agent Spend Its Next Tokens? Causal Attribution of Inference-Time Resources

1 Introduction

• End-to-end comparisons confound resource, state, and revision changes, while one-at-a-time ablations and incomplete combinations cannot distinguish bottlenecks from interactions.With bounded outcomes, apparent interactions may reflect substitution, complementarity, or a ceiling.Source: §1 · p. 1
• BRA assigns all combinations of computation, information, and feedback to replicated failed checkpoints, then uses a common revision and held-out scoring to measure marginal values and interactions.It averages each resource’s marginal contribution over bundles of the other resources and computes the resulting quantities exactly per checkpoint and model.Source: §1 · p. 2
• Task-relevant specifications improve held-out utility across models and task families, while public-test feedback adds at most 0.03 and retrieved code adds little or none.The value of computation declines with model capability, while information remains useful; resource channels mostly show diminishing returns, although information components can complement one another.Source: §1 · p. 2
• Three of four frontier models almost always request feedback and realize lower repair gains than uniform random resource selection.The paper reports that the fixated models request feedback in 95–100% of cases.Source: §1 · p. 2

2 Preliminaries

• A frozen failed checkpoint is assigned every bundle of the three resources in a full factorial, making all interactions estimable without assumptions.The bundle’s expected utility defines its worth; the gain game measures utility above the empty-resource bundle.Source: §2 · p. 2
• A resource’s Banzhaf value averages its marginal contribution over bundles of the other resources, while interaction indices indicate complementarity or substitution.Negative interactions can also arise from a utility ceiling, so they are interpreted relative to the empty-bundle outcome’s distance from that ceiling.Source: §2 · p. 3, Figure 1 · p. 3
• Allocation regret compares a policy’s selected resource gain with the checkpoint-wise oracle’s best feasible resource gain.With additive costs, each resource’s return is its marginal contribution divided by its cost.Source: §2 · p. 3

3.1 Checkpoint-conditioned counterfactual forks

• BRA clones each failed checkpoint across all eight resource combinations, applies an identical final revision, and scores each artifact on held-out tests.The full factorial identifies every interaction; varying only the assigned resources makes cell differences attributable to the preceding interventions.Source: §3.1 · p. 4, §3 · p. 4

3 Banzhaf Resource Attribution (BRA)

• The intervention definitions and contrasts were frozen before outcomes were inspected, with controls separating computation, information, and feedback channels.Only the final revision can change the artifact; held-out inputs, expected outputs, and reference solutions are excluded from prompts.Source: §3.2 · p. 4, §3.4 · p. 5

3.2 Three resources, orthogonal as interventions and graded as doses

• Computation is an analysis-only planning call, information is an artifact-independent task packet, and feedback reports execution results without modifying the artifact.Information packets range from repository retrieval to location, behavior, and specification; feedback detail ranges from pass counts to observed outputs. The interventions are defined separately by channel, although the resources are not orthogonal in nature.Source: §3.2 · p. 4

3.3 Estimands, fingerprint, and inference

• The seven-contrast bottleneck fingerprint records average resource effects and interactions; standalone sufficiency and conditional necessity distinguish a resource’s value alone from its value in the full bundle.Checkpoint-level contrasts are computed from rollout cell means, with checkpoint resampling for confidence intervals and Holm correction for the three pairwise interactions.Source: §3.3 · p. 5

3.4 Comparing models and scoring the agent’s own allocation

• BRA compares models on shared failed states as well as their own failures, and scores a model’s single-resource choice against always-selecting a resource and uniform random selection.The choice probe permutes option order to distinguish preference from position.Source: §3.4 · p. 5

4 Experimental Setup

• The evaluation spans six models and two task families, with 194 function-level failures and 126 repository-level failures.Function-level checkpoints are first-attempt failures by Qwen3-4B on MBPP+ and HumanEval+; repository-level checkpoints are unresolved SWE-bench Verified patches.Source: §4 · p. 6
• Function-level utility is the fraction of held-out extended tests passed, while repository-level utility is the fraction of held-out FAIL_TO_PASS tests passed, with regressions setting utility to zero.The repository sample includes 126 unresolved patches from a stratified frame of 250 instances.Source: §4 · p. 6
• The treatment design varies planning-call budgets, feedback detail, and information packets, with rollout counts and revision budgets differing by task family.Function-level information uses a specification packet and placebo; repository-level main runs use retrieval, with additional information rungs in dose arms.Source: §4 · p. 6

5.1 Q1: Information is the binding resource; feedback is not

• 0.064 (GPT-5.6) to 0.136 (Qwen3-14B) are the function-level information effects; repository specifications yield 0.048–0.103, while retrieved code and feedback remain near zero.The repository comparison replaces BM25-retrieved context with an audited specification; Table 1 reports the fixed-state panel.Source: §5.1 · p. 6, Table 1 · p. 7, §5.4 · p. 8

5.2 Q2: Capability moves the bottleneck from computation to information

• Computation effects fall from 0.131 along the Qwen3 ladder to 0.010 and 0.007 for GPT-5.6 and Sonnet 4.5, while information remains largest for five models.On the 24 jointly difficult checkpoints, information remains significant for every model; computation is not rescued for GPT-5.6 or Sonnet 4.5.Source: §5.2 · p. 7, Figure 5 · p. 9

5.3 Q3: Resources substitute for one another

• 21 of 36 pairwise interactions are significantly negative after Holm correction and none is significantly positive, indicating that resources generally substitute.Away from the ceiling, computation–information interactions are −0.10 to −0.14 and significant for every model except Qwen3-4B.Source: §5.3 · p. 7

5.5 Q5: Models ask for the wrong resource

• Three models request feedback in 95–100% of checkpoints, yet at function level their choices realise only 10–27% of oracle gain and perform worse than random selection.Always requesting information beats each of those three models on function-level failures; on repositories it beats all four models, significantly for three.Source: Figure 4 · p. 8, §5.5 · pp. 8–9

5.4 Q4: On repositories, retrieval is not information

• 0.013–0.047 is the effect of knowing what a repository fix must do; knowing only where it goes is worth −0.03 to +0.02, while both together yield 0.048–0.103.With retrieved code, repository information effects are near zero; audited specifications instead produce positive effects across the reported models.Source: §5.4 · p. 8

5.6 Q6: Return per cost, non-fungibility, and greedy allocation

• Per 1,000 added tokens, information returns 0.54–1.25 at function level and 0.11–0.31 on repositories; greedy allocation reaches 0.80–0.99 and 0.89–0.97 of optimal gain, respectively.The resource indicators predict cell means better than total token cost for every model and task family except a tie for GPT-5.6 at function level.Source: §5.6 · p. 9

A Discussion and Conclusion

• BRA’s fixed-state, single-revision measurements compare resource bundles on identical failures, and the resulting surfaces can score allocation against a per-checkpoint optimum.The authors frame the oracle-to-own gap as a bound on what a learned allocator could recover.Source: A Discussion and Conclusion · p. 13

Limitations

• BRA measures feedback as information for one revision, not its value as a verifier or termination signal across an agent trajectory.The checkpoints come from failures of one weak generator and one weak agent; the within-family capability ladder removes vendor confounding only up to 32B.Source: Limitations · p. 13

C.1 Proofs of Proposition 1 and Shapley values

• Each Banzhaf value averages a resource’s marginal contribution over the four bundles of the other resources; pairwise interactions average second differences, while the three-way interaction is the alternating sum over all eight cells.For three resources, the pairwise Banzhaf and Shapley interaction indices coincide.Source: §C.1 · p. 14
• The fingerprint together with q000 is an invertible linear transform of the eight cell means, while nonpositive average pairwise interactions alone do not establish submodularity.Submodularity requires every pairwise second difference to be nonpositive at every level of the third resource.Source: §C.1 · p. 15, §C.2 · p. 15

C.2 Greedy Guarantees

• For monotone submodular utility with additive costs, greedy selection plus the best single element guarantees at least (1 − 1/e)/2 of the budgeted optimum; partial enumeration raises the bound to 1 − 1/e.The paper uses these guarantees as motivation and reports the realised greedy-to-optimal ratio directly.Source: §C.2 · p. 15

D Setup details

• Function-level checkpoints are failed first attempts by Qwen3-4B, while repository checkpoints are unresolved, nondegenerate patches from a stratified SWE-bench Verified frame.Specifications were audited against reference-code and test leakage; artifacts were scored with EvalPlus or the SWE-bench harness under the stated evaluation setup.Source: D Setup details · pp. 15–16

E Repository-level edit protocol

• Figure 6 reports function-level label prevalence at τ = 0.2 and ceiling 0.8, distinguishing checkpoints responsive to one resource from those responsive to several.The caption defines those categories by responsiveness to exactly one or more than one single resource.Source: Figure 6 · p. 17
• Figure 7 plots computation-only and feedback-only dose curves on 40 hardest function-level checkpoints, with paired-gain 95% checkpoint-bootstrap intervals around the mean.The caption identifies DeepSeek-V3.2, k = 5, and empty-cell q000 = 0.298.Source: Figure 7 · p. 17
• Revision blocks are matched in a fixed fallback order, and partially applicable responses are applied and scored while block outcomes are recorded.Responses with no applicable block score 0; edits reverting files to the base commit are scored as that base commit.Source: E Repository-level edit protocol · pp. 16–17

F Deviations from the pre-registered protocol

• After shakedown, the planning budget rose to 2048 tokens, repository edit matching gained fallbacks, and revision-caused time-outs were scored as failures.The ledger also records the information-dose ladder and repository oracle arm as additions after the main runs.Source: F Deviations from the pre-registered… · pp. 17–18
• The repository-level robustness table reports interactions under retrieval and specification arms, main effects by artifact stratum, and a native-failure panel.The repository panel contains 126 checkpoints.Source: Table 6 · p. 18
• The checkpoint generator changed to Qwen3-14B because DeepSeek-V3.2 resolved most instances within 75 steps; its 30 unresolved patches became the native-failure panel.Qwen3-32B was served through a cloud API, and the behaviour-only audit was relaxed to permit names needed to describe behaviour.Source: F Deviations from the pre-registered… · pp. 17–18

G Compute

• Qwen3-4B and Qwen3-14B ran locally on a node with three NVIDIA A100 40 GB GPUs, 96 CPU cores, and 1.1 TB of memory.All other models were called through APIs.Source: G Compute · p. 18
• Evaluation ran on the same node, using sandboxed CPU workers for EvalPlus and benchmark Docker images for the SWE-bench harness.Source: G Compute · p. 18
• A function-level fixed-state panel took one to two hours and a repository-level arm two to nine hours; harness evaluation dominated repository runtime.The function-level panel comprised 194 checkpoints and eight cells; the repository arm used k = 3.Source: G Compute · p. 18

• Bold entries use two-sided checkpoint-bootstrap tests, with Holm adjustment for the three pairwise interactions within each model and no adjustment otherwise; bracketed ranges are 95% intervals.Source: H Additional results · p. 18

H.1 Function level

• Information alone is the best single cell for every model except Qwen3-4B, where computation scores 0.499 against 0.418 for information; full-bundle utility reaches 0.60 for Qwen3-4B and at most 0.91.Source: §H.1 · p. 18, Table 8 · p. 19
• For GPT-5.6 and Sonnet 4.5, information-only utility is within 0.01 of the full bundle, feedback removal has no significant cost, and information removal matters for every model.Source: §H.1 · p. 19
• Away from the ceiling, computation shrinks for the three Qwen models, while information effects grow on the matched subset; subset selection inflates main effects and pushes interactions down.These checks test whether ceiling effects explain the pattern, not the exact effect sizes.Source: §H.1 · p. 19
• Feedback alone unlocks at most 1.5% of checkpoints for any model; computation dominates single-resource unlocks for Qwen3-4B, while information does so for GPT-5.6 and Sonnet 4.5.Source: §H.1 · p. 19
• Computation saturates early: 256 tokens yield 61% of the 2048-token gain, while 4096 tokens add nothing; more detailed repository feedback brings no detectable gain on the tested subset.The specification-minus-nothing contrast exceeds the average information effect because the average includes bundles where other resources have already recovered some failures.Source: §H.1 · p. 19

H.2 Repository level

• On repository tasks, specification effects are positive for every model, and no specification-arm interaction is significant or exceeds 0.033 in absolute value.Effects are similar across artifact strata for DeepSeek-V3.2, GPT-5.6, and Sonnet 4.5 (0.066–0.108).Source: §H.2 · p. 19

• Under the binary all-tests-pass outcome, information remains the largest effect where it was under utility, while computation vanishes for GPT-5.6 and Sonnet 4.5 at function level.Source: §H.3 · pp. 19–20, Table 9 · p. 20
• Thinking-mode computation yields AMEC = 0.120 against 0.110 for planning, but exhausts its budget without code in 95 of 2328 cells, scored zero under the fixed-budget rule.Source: §H.3 · p. 20
• Neither DeepSeek-V3.2 nor GPT-5.6 gains from repository planning beyond 2048 tokens in the reported 4096- and 8192-token checks.Source: §H.3 · p. 20
• On 33 split-test instances, fuller feedback changes DeepSeek-V3.2’s feedback-only and retrieval-plus-feedback cells by −0.008 and −0.043, respectively, with no detectable gain.Source: §H.3 · p. 20
• The clean-edit subset is a sensitivity check, not an estimate; too few checkpoints remain for reliable estimates for DeepSeek-V3.2 and Qwen3-14B.Source: §H.3 · p. 20
• Across option orders, the three fixated models still request feedback as their majority choice, despite the reported preference-result ranges varying by model.Source: §H.3 · p. 20