PaperlayerView exact revision on arXiv

Source-linked AI summary

Coverage, Not Credit: Failure-Credit Routing of Zeroth-Order Perturbation Budgets Does Not Improve On-Pool Sample Efficiency for LLM Agents

Yuxu Ge

arXiv:2608.28011v1cs.AI

TL;DR

The paper asks whether trajectory-level failure credit should route fixed ZO/ES perturbation budgets among modules of frozen LLM agents. Across multiple agents, tasks, allocation schemes, and controls, routing produced no detectable on-pool optimization benefit over uniform coverage, while held-out BFCL endpoints showed one positive exception.

  • Problem

    The paper examines whether localized, verifiable failure credit should determine where fixed ZO/ES perturbation budgets are spent when optimizing tool-using agents with frozen LLMs.

  • Method

    The study compares six budget-allocation schemes across synthetic and BFCL-derived tasks, frozen Qwen2.5 and SmolLM2 agents, credit-noise conditions, paired seeds, sign-flip tests, coverage floors, and held-out BFCL endpoints.

  • Results

    No statistically detectable improvement over uniform allocation appeared in any on-pool comparison, while soft routing exceeded uniform by +0.047 on held-out BFCL endpoints (p = 0.031, n = 6).

  • Takeaways & Limitations

    Uniform coverage, or routing with a per-module floor, is the robust default for on-pool optimization because concentrating perturbations can interrupt another module’s cumulative movement.

  • Takeaways & Limitations

    The primary estimand is optimization efficiency on a fixed task pool; held-out evaluation covers endpoints from one task family, and held-out learning curves were unavailable.

Abstract

from arXiv · show

Trajectory-level credit assignment can localize which module of a tool-using LLM agent causes failures using only verifiable signals. We ask whether such failure credit should route a fixed zeroth-order/evolution-strategies (ZO/ES) perturbation budget. Across a synthetic environment and frozen Qwen2.5-1.5B/3B and SmolLM2-1.7B agents, three task families, six allocation schemes, a credit-noise sweep, paired seeds, and exact sign-flip tests, we find no statistically detectable improvement over uniform allocation in any on-pool comparison (no gain of at least 2 percentage points). The joint soft-plus-sigma scheme is equivalent to uniform within a +/- 0.02 AUC margin on 1.5B and 3B; concentrating the full budget on the credit argmax is marginally equivalent on 1.5B, where that module is the verified bottleneck, and significantly worse on 3B. Inverse-propensity debiasing does not rescue routing, and misrouting costs up to -0.074 AUC in-house and -0.118 end-to-end on the BFCL-derived family. Across six fixed-step schedules, loss is linear in bottleneck starvation rate (R^2 = 0.94, descriptive), and a preregistered credit-free coverage floor removes detected harm. Matched-budget burst and step-compensating catch-up schedules are consistent with harm arising from insufficient cumulative parameter movement rather than update frequency. Our primary estimand is optimization efficiency on a fixed task pool. On unseen BFCL functions, the study's one exception is that soft routing exceeds uniform on held-out endpoints (+0.047, p = 0.031, n = 6). A plausible but untested reading is that routing-favored caller improvements transfer while uniform's on-pool gains reflect a synthesizer behavior specific to our harness. We report this exception explicitly and document three failure modes that can silently invalidate ZO/ES experiments on frozen LLMs.

1 Introduction

The paper tests whether trajectory-level failure credit should route a fixed ZO/ES perturbation budget among agent modules. Across on-pool comparisons, routing shows no detectable efficiency benefit over uniform allocation, while coverage protects against starvation-related harm.

  • No statistically detectable improvement over uniform appears in any headline on-pool comparison.Soft routing is equivalent within ±0.02 on Qwen2.5-1.5B and 3B; concentrating 97% on the verified bottleneck is marginally equivalent on 1.5B but worse on 3B.
  • −0.074 AUC results from concentrating the budget on a random module.Increasing routing-signal corruption probability q ∈{0.25, 0.5, 0.75} monotonically degrades point estimates.
  • ∆AUC ≈−0.001 −0.087 · starvation, with R^2 = 0.94, across six allocation schemes.The regression is descriptive and relates loss to the probability that the true bottleneck receives zero pairs in a generation.
  • The preregistered credit-free floor recovered 80% of a nonsignificant misrouting penalty and was TOST-equivalent to uniform.The floor guarantees at least one pair per module per generation.
  • −0.075 harm under burst starvation and recovery under catch-up schedules are consistent with insufficient cumulative movement as a mediator.The matched-budget schedules keep update frequency from explaining the observed difference.
  • +0.047 held-out endpoint success favors soft routing on unseen BFCL functions.This is the study’s one exception to the on-pool result.
  • The robust practical default is uniform coverage or a per-module floor, while transfer benefits remain an open question.The tested claim is scoped to modular ZO/ES on frozen LLMs with tied low-dimensional subspaces; other parameterizations remain untested.
  • The study documents falsified intermediate conclusions and bases its final claims on a preregistered test.A concentration dose–response relationship and an analogy-based sparse-baseline prediction were later rejected by controls.

2 Related work

Prior work establishes ZO/ES fine-tuning, adaptive allocation, agent credit assignment, and subspace perturbations, but does not test routing parameter-space search budgets by trajectory credit. The paper positions its contribution as this controlled bridge.

  • ZO/ES methods enable memory-efficient fine-tuning of frozen LLMs when backpropagation is unavailable or memory-prohibitive.
  • Adaptive-ZO methods mainly report wall-clock gains of 1.7–4.5× and small accuracy deltas over uniform or full baselines.Reported deltas range from approximately +0.1 to +1.2 percentage points, though the metrics are not directly comparable here.
  • Existing agent credit methods route blame to textual or program updates, while cooperative ES decomposes parameters without credit-driven routing.
  • The paper reports no prior work routing parameter-space search budget by trajectory credit and tests whether this bridge improves on-pool optimization.
  • Classical ES uses antithetic mirrored sampling, while adaptive-subspace ES shapes perturbation distributions rather than routing budgets among modules.Randomized block-coordinate descent provides a related coverage principle requiring every block to be sampled with probability bounded away from zero.

3 Setup

The setup evaluates modular ZO/ES updates for frozen tool-using agents under verifiable rewards, shared randomness, and a fixed-task-pool efficiency estimand. Modules receive low-dimensional perturbations and allocation schemes determine their pair counts.

  • The agent has Planner, Selector, Caller, and Synthesizer stages, with environment-verifiable tool execution and exact-answer success.Caller retries up to twice with error feedback.
  • Each module owns a 32-dimensional coefficient vector applied through low-dimensional singular-vector subspaces.The parameterization targets q/v/o projections across model blocks and stores 128 coefficients.
  • Verifiable failures assign blame to Selector, Caller, or Synthesizer, while the Planner has no observable credit signal.An EMA tracker converts blame into normalized credit used by routing schemes.
  • Antithetic mirrored ES uses common random numbers, trust-region clipping, and a reward combining task success with teacher-forced log probability.
  • Paired seeds support exact sign-flip tests, and the primary metric is time-averaged success over identical per-generation rollout costs.Equivalence uses TOST with a ±0.02 margin.
  • The primary estimand is optimization efficiency on a reused fixed task pool rather than generalization to held-out examples.

4 Three silent failure modes

The paper identifies three silent ZO/ES failure modes: behaviorally inert random bases, divergence from unbounded steps, and piecewise-constant greedy rewards. Controls show that magnitude, trust regions, and likelihood shaping determine whether experiments produce interpretable learning signals.

  • T1: random subspace bases are behaviorally inert: Unit-norm random rank-1 bases produced zero behavioral change despite 40× scaling, making them an invalid basis for trusting a null result.
  • The study treats these issues as prerequisites for frozen-LLM ZO/ES experiments, not merely implementation details.
  • T1: random subspace bases are behaviorally inert: ∆AUC = +0.008, p = 0.75, n = 6 shows magnitude-matched random directions match SVD directions.Magnitude-matched random directions slightly exceed SVD at +0.035, p = 0.06.
  • T2: strong bases + unbounded steps diverge: Unbounded steps with strong SVD bases caused parameter-norm explosion and collapsed success, whereas a trust region removed the failure mode.Tightening the trust region from 0.5 to 0.125 costs −0.072 AUC by truncating informative steps.
  • T3: greedy decoding makes task reward piecewise-constant: Greedy decoding can make mirrored ES estimates identically zero when perturbations do not flip output tokens.A verifiable log-likelihood shaping term restores a followable gradient under weak bases; with strong SVD bases, pure task reward suffices.

5 Results

Across synthetic and frozen-LLM experiments, routing perturbation budgets by failure credit did not improve on-pool optimization over uniform allocation. The results instead identify bottleneck coverage and cumulative parameter movement as the relevant safeguards, while held-out transfer provides one explicit exception.

  • 5.1 Synthetic pre-study: Oracle routing never outperformed uniform in four synthetic regimes, while wrongly concentrated budgets cost up to −0.220 AUC.All oracle differences were ≤0.009 AUC; the minimum reported p-value was 0.11.
  • 5.2 ES works; routing does not help: Uniform improved the frozen Qwen2.5-1.5B agent to ≈0.92 success, while only the misrouted arm lagged.The final uniform generation reached 0.945.
  • 5.2 ES works; routing does not help: Soft routing was equivalent to uniform within ±0.02 on Qwen2.5-1.5B, while hard versus random differed by +0.0714 AUC.The soft result comes from the preregistered 12-seed replication; the hard-versus-random contrast was significant across 8/8 seeds.
  • 5.2 ES works; routing does not help: The apparent concentration dose–response disappeared after testing a position-correct triple, showing that concentration and routing position had been confounded.The slope changed from −0.099 with uniform/soft/random to −0.005 with uniform/soft/hard.
  • 5.2 ES works; routing does not help: On Qwen2.5-3B, soft remained equivalent to uniform, random was worse by −0.055 AUC, and hard was worse by −0.027 AUC.Hard beat random directionally in 5/6 seeds, but the comparison was not significant.
  • 5.7 The starvation regression: Across six allocation schemes, loss tracked bottleneck starvation linearly, with slope −0.091 on measured rates and R2 = 0.973.The reported coverage interpretation is that zero-starvation schemes pay nothing, while the linear coefficient remains exploratory.
  • 5.8 Preregistered test of the coverage floor: A preregistered coverage floor recovered 80% of a hard∼0.5 penalty and made the floored variant TOST-equivalent to uniform.The floor guaranteed at least one pair per module per generation; the in-house penalty itself was not significant at n = 6.

5.9 Shaping-term robustness

Robustness checks preserve the central pattern: routing does not improve on-pool optimization, while concentrated or starved updates can harm performance; coverage and module movement matter more than routing signal quality.

  • −0.118 end-to-end performance marked hard routing’s BFCL penalty, while adding a coverage floor shifted it by +0.133 and removed the detectable difference from uniform.
  • On unseen BFCL functions, soft routing beat uniform by +0.047, but this was the study’s single held-out exception rather than an on-pool optimization gain.
  • −0.072 was the loss from capacity-matched uniform routing, showing that equalizing total clip capacity did not explain the arm ordering.
  • Realized total movement, rather than total capacity, tracked outcomes: uniform moved 0.37 per generation, hard 0.12, and random 0.08.
  • Inverse-propensity debiasing did not rescue soft routing, and basis-binding, basis-magnitude, and matched-budget burst checks left the qualitative ordering intact.
  • Uniform allocation was never significantly outperformed in any on-pool comparison across the study’s three task families and six allocation schemes.

6 Discussion and limitations

The discussion interprets coverage as the robust default for the tested frozen-LLM regime, while limiting the conclusion to fixed-pool optimization and identifying transfer, scale, and multiple-testing boundaries.

  • Discussion: Extra perturbation pairs mainly reduce variance, whereas cumulative module displacement is the binding resource that uniform coverage supplies.
  • Discussion: The held-out BFCL exception suggests routing may help when the credit-targeted caller is also the module whose improvements transfer, but this remains untested.
  • Limitations: The primary estimand is optimization efficiency on a fixed task pool; held-out evidence covers endpoints from one task family, with no held-out learning curves.
  • Limitations: The coverage prescription may not directly execute in adaptive-ZO regimes with tens to millions of units, where per-unit floors are infeasible.
  • Limitations: The study reports roughly seventy significance tests without family-wise correction, making floor-level findings vulnerable to chance under a global null.
  • Takeaway: The recommendation is to choose subsets for systems reasons, preserve coverage where budgets permit, and not expect credit signals to accelerate convergence.

7 Conclusion

The study finds that routing ZO/ES budget by trajectory-level failure credit provides no on-pool optimization-efficiency gain and can impose downside when it interrupts another module’s cumulative movement. Uniform coverage and per-module floors are therefore the robust defaults tested, while transfer-oriented routing remains unresolved.

  • 7 Conclusion: On-pool credit-based routing bought nothing in optimization efficiency and carried downside when it interrupted another module’s cumulative movement.The authors frame whether routing toward modules whose gains transfer can improve generalization as an open question.
  • 7 Conclusion: Uniform coverage—or a per-module floor in the tested routing families—is the robust default for optimizing on a task pool.The conclusion treats the generalization possibility suggested by one held-out result as unsettled rather than established.
  • 7 Conclusion: The anonymous supplement includes code, task files, preregistration records, and per-seed, per-generation data sufficient to recompute every statistic and figure.Randomness is derived from sha256-keyed streams, and interrupted runs resume bit-identically.

A Preregistration ledger

The preregistration ledger records that predictions were specified before outcomes were reported, with prediction results and diagnostic controls documented in the experimental record.

  • A Preregistration ledger: Every prediction was written before the commit that reported its outcome, with commit order documented by a hash/date/subject manifest.The public repository history is scheduled for release after review.
  • A Preregistration ledger: Nineteen of twenty-eight scored predictions hit, while one inert manipulation was recorded as a control.A separate follow-up was specified as a two-sided diagnostic before it ran.

B Starvation regression: raw points

The raw regression points come from six paired in-house sweeps and support a strong descriptive relationship between bottleneck starvation and AUC loss.

  • B Starvation regression: raw points: Six regression points come from in-house sweeps at n = 8, each paired against the same uniform seeds.The regression uses the analytic nominal starvation rate, with measured allocation-log rates shown for audit.
  • B Starvation regression: raw points: ∆AUC = −0.001 −0.087 · starvation, with R2 = 0.941.The fit is based on the analytic nominal starvation rate.

C Synthetic pre-study: the four regimes

In the deterministic synthetic environment, knowing the true bottleneck does not improve performance over uniform allocation, whereas concentrating budget on a random module can be substantially harmful.

  • C Synthetic pre-study: the four regimes: Oracle routing shows no detectable advantage over uniform in any synthetic regime, with |∆| ≤0.009.The comparison uses 10 paired seeds and 120 generations per regime.
  • C Synthetic pre-study: the four regimes: Wrongly concentrated random routing costs up to −0.220 in the deterministic synthetic environment.Deltas are measured versus uniform across the four regimes.

D Experiment overview

The experiments combine paired, parameterized sweeps across backbones, task pools, allocation arms, seeds, and synthetic regimes, while preregistration and starvation-regression tables document the evaluation record. The overview also distinguishes scored outcomes, inert controls, and the fitted relationship between starvation and AUC.

  • Protocol: Unless stated otherwise, experiments use Qwen2.5-1.5B-Instruct with 32-dimensional SVD-basis modules, σ = 0.3, learning rate 0.3, and greedy decoding.Evaluation batches are 16, or 12 on BFCL; every sweep is paired over listed seeds.
  • Preregistration: The preregistration ledger records 28 scored outcomes: 19 hits and 9 misses, with one inert control and one unscored row.The significant loss was unpredicted, and the sparse prediction was an informal analogy rather than a separately time-stamped preregistration.
  • Starvation analysis: Across starvation-regression points with n = 8 each, nominal and measured starvation rates agree within 0.05, and measured-rate refitting gives slope −0.091 and R2 = 0.973.The regression uses paired sign-flip p-values.
  • Synthetic pre-study: In the synthetic pre-study, oracle versus uniform passes the ±0.02 TOST equivalence margin in three of four regimes, while the non-stationary/tight regime is uncertified.The four regime-specific TOST p-value pairs are reported in row order in Table 6.
  • Task pools: BFCL-48 is the 48-task training pool, while BFCL-96 unseen uses disjoint questions and function names; synthetic regimes vary pair counts and module dimensions.Synthetic regimes use 4 or 8 pairs and module dimensions 16, 8, 16, and 32 in Table 6’s row order.
Loading 2608.28011v1…