PaperlayerView exact revision on arXiv

Source-linked AI summary

Marginally Correct Tool Caches Can Reverse Group-Normalized Policy Updates

Shivam Gupta

arXiv:2609.26866v1cs.LG

TL;DR

Group-relative training can depend on how stochastic tool outcomes are coupled across rollouts, even when each rollout retains the same conditional reward distribution. The paper analyzes shared execution in a two-action model and finds that it can reverse the expected normalized update; the result is bounded to the studied setting.

  • Problem

    Does preserving each rollout’s conditional reward distribution suffice to preserve the expected group-normalized update when a cache shares stochastic tool outcomes?

    Source: §1 · p. 1

  • Method

    The paper characterizes finite-group shared-outcome updates for scalar rewards and compares them with independent execution and a centered estimator.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    At G=64, independent normalization gives −0.078867 while sharing gives +0.297628, although both laws have mean reward 0.85 and true gradient −0.025.

    Source: §5.2 · p. 5

  • Takeaways & Limitations

    Marginal output validity alone cannot certify stochastic caching as training-equivalent; evaluate reuse against its estimator and measure physical tool outcomes separately from rollout count.

    Source: §8 · p. 7

    The analysis uses a fixed-policy, two-action, one-decision model and does not establish language-model training performance or end-to-end convergence.

    Source: §7 · p. 6

Abstract

from arXiv · show

Tool-result caching reduces repeated execution in agent training, but also couples rollout randomness. We study a two-action model in which independent and shared execution preserve every rollout's conditional reward distribution. Despite this marginal agreement, sharing one stochastic result per group can reverse the expected group-normalized policy update. We derive an exact finite-group expression: against a constant alternative, the shared update follows the probability of winning minus the probability of losing, rather than the difference in expected reward. A Bernoulli specialization yields a wrong-direction region and a non-vanishing update-variance floor as group size grows. Centering without group standard-deviation scaling preserves the expected-return direction in this model, using an existing estimator control. Exhaustive finite sums verify 540 configurations and 3,240 estimator evaluations, with a separate ordered-sequence checker. An implementation audit reproduces the sharing path in a pinned, unmodified TVCache stack using 256 scripted rollouts. These results do not measure language-model training performance or refute TVCache's deterministic-output contract. They establish that marginal output validity alone cannot certify a stochastic cache as training-equivalent.

1 Introduction

The paper asks whether preserving each rollout’s conditional reward distribution is enough to preserve a group-normalized update. It shows that shared stochastic outcomes can reverse that update and outlines exact, computational, and implementation checks of the effect.

1 Introduction

  • Sharing one stochastic result preserves individual callers’ conditional reward distributions but changes the group’s joint distribution, which a within-group learner can distinguish.Independent calls draw several outcomes; shared execution draws one and repeats it.Source: §1 · p. 1
  • For Bernoulli payoffs, sharing can set the preference threshold at 1/2 even when the alternative has a substantially higher expected payoff.The shared normalizer removes the realized payoff gap’s magnitude, leaving its sign; the cache is refreshed for every group.Source: §1 · p. 1
  • The paper studies how controlled changes in tool execution induce established group-normalization bias, rather than claiming that the bias or centered alternatives are new.Group-normalization bias and centering-based alternatives are established [2] [3].Source: §1 · p. 1
  • A finite-group characterization covers arbitrary scalar rewards, while a Bernoulli example demonstrates sign reversal and separates marginal reward agreement from update agreement.The analysis also identifies asymptotic means, a persistent variance floor, and an existing centered-estimator control.Source: §1 · p. 1
  • Exhaustive numerical checks and a pinned runtime audit support reproducibility, but the work does not measure production prevalence, model accuracy loss, or monetary savings.The results identify a condition to test when optimizing expensive stochastic tools.Source: §1 · p. 2

2 Related work and scope

The paper situates its analysis among work on group-relative optimization, reward normalization, and tool caching. It distinguishes stochastic sampling-law effects from state consistency and limits claims about prior systems and conceptual novelty.

2 Related work and scope

  • GRPO uses group-relative advantages [1], while Dr. GRPO addresses biases associated with group-level standard-deviation scaling and response-length normalization [2].The paper’s centered-only control uses this existing principle and analyzes a one-step on-policy score estimator, not the complete clipped, multi-epoch, token-weighted algorithm.Source: §2 · p. 2
  • The Bernoulli independent-execution model is affinely equivalent to Che et al.’s three-reward setting [3], limiting claims of conceptual novelty.Sharing makes the two-level reward structure arise at every interior action probability, rather than only in a sparse-action regime.Source: §2 · p. 2
  • Stateful cache correctness does not by itself establish joint reward-law equivalence; the paper’s stochastic setting is outside TVCache’s deterministic-output contract [6].The paper neither reproduces prior training results nor claims their reported benchmarks exhibit this effect.Source: §2 · p. 2

3 Model and estimators

The model compares two actions, one with constant reward and one with a random reward, under independent or shared tool execution. It defines a within-group reward-standardized update and bounds its scope to an on-policy reward component.

3 Model and estimators

  • Action A returns constant c, while action B returns scalar reward Y with finite first moment μ; rollouts independently choose actions with probability p for B.Actions are independent of tool randomness.Source: §3 · p. 2
  • The sampled-action policy score is a_i − p, and the normalized estimator divides the centered score–reward sum by within-group population reward standard deviation.When that standard deviation is zero, U0 is defined as zero; positive updates increase the probability of B.Source: §3 · pp. 2–3
  • Independent execution gives each B invocation an independent reward draw, whereas shared execution reuses one group-level draw with no persistence between groups.The intervention preserves conditional reward marginals; B outcomes are independent across rollouts in one mode and identical in the other.Source: §3 · p. 2
  • These estimators cover only the reward component of an on-policy update at unit importance ratio, not KL penalties, clipping, adaptive optimizers, variable token lengths, or multi-turn feedback.The model does not include those training components or feedback effects.Source: §3 · p. 3

4 Exact consequences of sharing

With shared execution, the normalized update can track whether the stochastic payoff beats a constant alternative rather than the expected-payoff advantage. Bernoulli rewards yield a wrong-direction region and a persistent within-group variance floor, while centering alone preserves the expected-gradient direction in this model.

4 Exact consequences of sharing

  • For ε=0, the shared update’s expected direction follows the probability of beating the constant alternative minus the probability of losing, not the expected-payoff advantage.The result concerns update direction, not whether an arbitrary full learning algorithm maximizes a median objective.Source: §4 · p. 3
  • For Bernoulli rewards, every interior p and G≥2 has a sign reversal when 1/2 < q < c < 1 or 0 < c < q < 1/2.The shared-update threshold is q=1/2, whereas the expected-return threshold is q=c; at G=2 both execution modes have the same expected update.Source: §4 · p. 3
  • For either execution law and integrable rewards, centering without group-standard-deviation scaling preserves the mean-gradient direction.The result is a direction guarantee for the centered estimator, not equivalence of the full update distribution.Source: §4 · p. 3
  • As G grows in the Bernoulli case, the shared update retains a nonzero within-group variance floor, although independent groups can average down this noise.In wrong-direction regions, sufficiently large independent and shared groups point in opposite directions.Source: §4 · p. 4

4.1 Nonzero numerical stabilization

  • With ε>0, the shared Bernoulli update’s zero-update threshold lies between 1/2 and c, approaching 1/2 as ε↓0 and c as ε→∞.A nonzero stabilizer therefore does not automatically restore the expected-payoff threshold.Source: §4.1 · p. 4

5 Exhaustive numerical verification

The authors verify the update formulas with exhaustive finite sums, numerical controls, and an independently implemented ordered-outcome checker. In a preselected witness, sharing reverses the expected normalized update despite unchanged conditional reward marginals.

5.1 Protocol and controls

  • The sweep evaluates 540 configurations and 3,240 estimator evaluations using finite probability-weighted sums, not Monte Carlo observations.It tests both execution modes with centering alone and normalization at ε∈{0, 10−4}.Source: §5.1 · p. 4
  • A separate ordered-rollout checker tests 288 small-group combinations, and all six numerical test methods pass.Maximum probability-mass error is below 3.34 × 10−15 and maximum marginal-mean error below 1.78 × 10−15; these are not statistical confidence intervals.Source: §5.1 · p. 4

5.2 A preselected witness

  • At G=64, independent normalization gives −0.078867 while sharing gives +0.297628; the centered estimator gives −0.024609 under either law.Both execution laws have mean reward 0.85 and true gradient −0.025; Table 1 reports the exact finite-sum witness.Source: §5.2 · p. 5, Table 1 · p. 5
  • As rollout count increases, the witness’s two normalized-update curves separate, while centering alone preserves their expected direction.The figure uses different vertical scales and shows exhaustive calculations, not learning curves or measured model performance.Source: Figure 1 · p. 5
  • Across the 540-configuration grid, execution modes have opposite normalized-update signs in 54 configurations at each tested ε; centering alone has no such discrepancy.The grid fractions are not deployment-prevalence estimates, and ε=10−4 checks robustness to one practical stabilizer only.Source: §5.2 · p. 5

6 A pinned implementation audit

The pinned audit exercised TVCache’s sharing path with scripted rollouts and found that shared execution repeated the first result from a varying tool stream. The probe establishes an implementation path for sharing, not a failure of the vendor’s deterministic-output contract or a measured training regression.

6.1 Execution boundary

  • The probe used unmodified stateful-executor, async-client, Flask-route, and immutable prefix-tree components, with only HTTP transport replaced by an in-process bridge.A scripted environment implemented the tool interface, and socket connections and DNS resolution were disabled.Source: §6.1 · pp. 5–6
  • Each case used 32 sequential one-call rollouts alternating A and B, comparing direct execution, shared identity, separate task namespaces, and separate draw identifiers.The varying B fixture alternated 1 and 0 on physical execution; it was not a Bernoulli sample.Source: §6.1 · p. 6, Table 2 · p. 6

6.2 Observed behavior and interpretation

  • 256 scripted rollouts across eight cases satisfied the protocol, with 196 physical tool calls, 648 in-process HTTP requests, and zero socket or DNS attempts.Three separate runs produced byte-identical result JSON; all 29 exported source files passed hash verification before import.Source: §6.2 · p. 6
  • The shared case executed only one A call and one B call, repeating the first B result in the varying fixture; identity-separated controls retained physical executions.Those controls trade away reuse and are not a cost-free fix.Source: §6.2 · p. 6
  • The inspected video integration includes a model-backed caption call, but its variability was not measured and the paper does not infer that it uses the group-normalized estimator.TVCache lists importance sampling for EgoSchema training and GRPO for other workloads.Source: §6.2 · p. 6

7 Implications, limitations, and reproducibility

The implications depend on the estimator: marginal tool-output agreement does not guarantee the same expected update under group standard-deviation normalization. The paper distinguishes this model-based finding from cost, deployment, and language-model training claims.

7 Implications, limitations, and reproducibility

  • Cache equivalence depends on the consuming estimator: sharing preserves the expected gradient for the mean-centered estimator in this model, but need not for the normalized estimator.Matching response means or individual output distributions is weaker than matching the expected update.Source: §7 · p. 6
  • The paper reports no time-to-accuracy or dollar-efficiency result; equal-rollout and equal-physical-call comparisons answer different questions.In the one-step Bernoulli model, independent execution requires Gp expected physical B calls per group, while sharing requires 1 − (1 − p)^G; the savings come with different gradient variance.Source: §7 · p. 6
  • The evidence is limited to two actions, one decision per rollout, a fixed policy, a constant alternative, and a specific on-policy reward estimator.The paper does not train or evaluate a language model, establish end-to-end convergence, or measure deployed stochastic-tool frequency.Source: §7 · p. 6
  • The artifact includes pre-execution protocols, an independent checker, finite-sum results, runtime traces, hashes, pinned dependencies, and figure-generation code.The reported experiments require no credentials, paid API calls, participant data, or model outputs to reproduce.Source: §7 · p. 7
  • A marginally valid cache can still change the learning problem, but this constructed counterexample does not show caching is generally unsafe or invalidate a named system’s published results.The stated intended use is auditing training optimizations before relying on efficiency claims.Source: §7 · p. 7

8 Conclusion

Sharing a stochastic tool result changes the joint sampling law seen by a group-relative learner and can reverse its expected normalized update despite identical conditional reward marginals. The conclusion is to evaluate stochastic reuse against the estimator it serves and measure physical outcomes separately from rollout count.

8 Conclusion

  • In the controlled model, shared execution can reverse the expected normalized update and leave a within-group variance floor as rollout count grows.The paper’s practical lesson is to evaluate stochastic reuse against its estimator and measure physical tool outcomes separately from rollout count.Source: §8 · p. 7

A Exact summation and equality controls

  • For the exact shared sum, the inner binomial distribution is replaced by mass 1 − q at k = 0 and q at k = n; when n = 0, only one outcome is counted.The numerical implementation stores update expectation and variance, probability mass, and expected reward for every estimator.Source: A Exact summation and equality controls · p. 8
  • At G = 2, deterministic controls q = 0 and q = 1 have identical update distributions under both execution laws, including zero centered updates for equal-action groups.Mixed-action groups have one B sample, so their update distributions also agree.Source: A Exact summation and equality controls · p. 8

B Variance of the centered estimator

The variance analysis conditions on the number of B actions and compares independent versus shared Bernoulli outcomes. It makes explicit that repairing the mean direction does not preserve gradient noise.

B Variance of the centered estimator

  • Conditioning on N and writing h = h_G(N), the analysis applies the law of total variance to the centered estimator.Source: B Variance of the centered estimator · p. 8
  • Sharing replaces the conditional variance Nq(1 − q) of independent Bernoulli outcomes with N^2q(1 − q), while the conditional means remain h(q − c).The squared coefficient is (G − N)^2/G^4; correcting the baseline factor multiplies either variance by G^2/(G − 1)^2.Source: B Variance of the centered estimator · p. 8
  • A mean-direction repair does not preserve gradient noise, so efficiency comparisons must account for independent groups and the physical-call budget.The paper does not turn these variance identities into a wall-clock or convergence claim.Source: B Variance of the centered estimator · p. 8

C Stabilizer threshold and general reward ties

The stabilizer threshold is bounded by the reward probability through strict termwise inequalities, while the general shared-reward result gives an exact second moment and variance extension. The runtime audit is limited to a narrow execution path and does not establish marginal agreement.

C Stabilizer threshold and general reward ties

  • For c>1/2, summing strict inequalities yields 1/2<qε<c; the inequalities reverse for c<1/2, and the threshold is 1/2 when c=1/2.Boundary action counts contribute zero, and finite sums allow both limits in ε to be taken termwise.Source: C Stabilizer threshold and general… · p. 8
  • The exact second moment is (1−1/G)p(1−p)Pr(Y≠c), and the variance extension subtracts S_G(p)^2[Pr(Y>c)−Pr(Y<c)]^2.A finite first moment is needed for comparison with expected return; the bounded normalized update is defined for any almost surely finite scalar Y.Source: C Stabilizer threshold and general… · p. 8

D Runtime audit boundaries

  • The runtime audit covers sequential one-call rollouts without stored forks, eviction, expiry, or production networking, and does not establish marginal agreement.Distinct identities are supplied by the experiment; its varying-output fixture is not the theorem’s probabilistic environment, so its returned means are not evidence of marginal agreement.Source: D Runtime audit boundaries · p. 8
Loading 2609.26866v1…