PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Action Credit Need Updating?

Hongye Yang, Boxiao Huang

arXiv:2609.29007v1cs.AIstat.ML

TL;DR

Policy updates can make historical action credit stale, but numerical value drift need not change the decision. The paper develops branch-sensitive transport and a gate to reuse, correct, or refresh credit; on an independent test set, the gate nearly matches a gap-based gate's regret while using fewer new tool steps.

  • Problem

    After a policy update, historical action credit may be stale, while recomputing every candidate from scratch requires additional tool calls and environment interaction.

    Source: §1 · pp. 1–2

  • Method

    The paper measures update effects on action-distinguishing branches, transports historical credit with old interventional trajectories, and uses DSC-Gate to choose reuse, transport, or refresh.

    Source: §1 · p. 2

  • Results

    On an independent test set, DSC-Gate substantially reduces new tool execution while preserving nearly identical regret to a gap-based gate.

    Source: §6 · p. 8, §7 · p. 9

  • Takeaways & Limitations

    Historical credit often remains useful after policy updates and can sometimes be corrected from existing trajectories before new execution is required.

    Source: §1 · p. 2, §7 · p. 9

    The experiments do not establish distribution-free finite-sample coverage or guarantees under arbitrary adaptive stopping, and external validation uses one model configuration and one tool environment.

    Source: §6 · p. 8

Abstract

from arXiv · show

Tool-using agents are continually updated with new interaction data. After each policy update, however, previously estimated action credits may become stale. Recomputing them from scratch can require many additional tool calls and environment interactions, making repeated updates increasingly expensive. We ask a simple question: when does historical action credit actually need to be updated? Our key observation is that a change in action value does not necessarily imply a change in the decision. Historical credit can still be useful as long as policy-induced drift is too small to overturn the existing action ranking. Building on this idea, we introduce pairwise branch sensitivity to capture how strongly a policy update affects the downstream regions that distinguish two candidate actions. We then derive a first-order anchored credit-transport estimator that updates historical credit using old interventional trajectories, and propose a Decision-Sufficient Credit Gate (DSC-Gate) that chooses whether to reuse, transport, or resample credit. Experiments show that branch sensitivity explains credit drift substantially better than global policy distance. With sufficient historical data, credit transport reduces estimation error, while its benefit to decision making is concentrated on updates that affect action-distinguishing branches. On a fully independent test set, DSC-Gate changes mean regret by only +0.00004 relative to a gap-based gate while reducing mean new tool steps from 472 to 286, a 39.4% reduction. We observe the same pattern after a real tool-agent parameter update. Overall, our results show that agents do not need to recompute action credit after every policy update: much of the historical evidence can be reused or cheaply corrected, reducing the additional interaction required to keep action decisions up to date.

1 INTRODUCTION

The paper asks when historical action credit becomes unusable after a policy update, arguing that value drift matters only when it can change the action decision. It proposes branch-sensitive credit transport and a gate that reuses, corrects, or refreshes evidence.

Motivation

  • Historical action credit can become stale as downstream behavior changes, making repeated from-scratch recomputation costly in tool calls and environment interaction.The introduction motivates the problem with a support agent choosing between routes such as checking order status or refund eligibility.Source: §1 · pp. 1–2
  • Whether credit needs updating depends on whether policy-induced drift can overturn the existing action comparison, not on global policy change or value drift alone.Updates to shared downstream regions may cancel between branches, while smaller changes to branch-differential regions can matter.Source: §1 · pp. 1–2

Contributions

  • The paper introduces first-order anchored credit transport and DSC-Gate, which uses historical trajectories to correct comparisons and selects reuse, transport, or new target-policy data.The gate combines the old gap, transported correction, and empirical uncertainty; new execution is reserved for unresolved comparisons.Source: §1 · p. 2

Experiments

  • Across controlled experiments, branch sensitivity explains credit drift better than global policy distance, transport helps when historical data are sufficient, and decision benefits concentrate on ranking-changing updates.On an independent test set, DSC-Gate changes mean regret by +0.00004 relative to a gap-based gate while reducing mean new tool steps from 472 to 286 (39.4%).Source: §1 · p. 2

Conclusion

  • The authors conclude that historical evidence can often be reused or corrected, reducing the additional interaction needed to keep decisions current as policies evolve.The validation includes a real tool-agent parameter update with the same conditional pattern.Source: §1 · p. 2

2 RELATED WORK

Related work establishes methods for reusing historical trajectories, acquiring evidence for comparisons, and studying tool agents under change. This paper focuses on whether historical evidence remains sufficient for a particular action comparison after a policy update.

Off-policy evaluation and historical-data reuse

  • Off-policy evaluation reuses trajectories after policy changes, while this paper asks the narrower question of whether old evidence still resolves a particular action comparison.Prior OPE work addresses target-policy value, finite-sample trust, unequal support, and estimator selection.Source: §2 · pp. 2–3

Pairwise comparison and adaptive evidence acquisition

  • Counterfactual learning and best-arm methods allocate evidence toward decisions, while this work adds the role of update location relative to shared and distinguishing action branches.The same global change can affect comparisons differently depending on whether it occurs in states shared by two branches or states that distinguish them.Source: §2 · p. 3

Tool-using agents under change

  • Prior tool-agent research studies multistep consistency, unstable tool preferences, and adaptation to changing tools or environments; this paper examines which old action comparisons remain usable after change.Source: §2 · p. 3

Selective and cost-aware interaction

  • Selective-interaction methods address avoiding unnecessary tool execution, and DSC-Gate applies that principle to deciding when new trajectories are needed for credit maintenance.The paper connects evidence reuse with pairwise decision sufficiency and execution cost.Source: §2 · pp. 3–4

3 CREDIT DRIFT AND DECISION SUFFICIENCY

The framework separates potential credit drift from whether that drift can change a decision. It estimates branch-specific effects, transports historical comparisons, and uses calibrated uncertainty to decide whether to reuse, transport, or refresh evidence.

3.1 PAIRWISE BRANCH SENSITIVITY

  • Pairwise branch sensitivity measures the policy update's signed overlap with visitation differences after forcing candidate root actions, so shared-region changes can cancel while branch-differential changes drive relative drift.Source: §3.1 · p. 4

3.2 FIRST-ORDER ANCHORED CREDIT TRANSPORT

  • First-order anchored transport uses old-policy trajectories to estimate a local policy-induced correction while retaining historical credit as its anchor.Its bias grows with omitted higher-order terms, and its variance depends on the amount and support of old data; an independently fitted baseline reduces variance without changing the first-order term's expectation.Source: §3.2 · p. 4

3.3 RANKING STABILITY

  • Ranking stability depends on branch sensitivity, the old action gap, and finite-data uncertainty, so numerical drift alone does not determine whether the selected action changes.Figure 2 presents 864 action pairs from 288 L2–L3 confirmation tasks per update condition and marks ranking reversals, with annotated rates also requiring a relevance threshold.Source: §3.3 · p. 4, Figure 2 · p. 5

3.4 DECISION-SUFFICIENT CREDIT GATE

  • DSC-Gate uses empirically calibrated radii to reuse historical credit when its gap is sufficient, transport when the corrected gap is separated from zero, and otherwise refresh through target-policy execution.With multiple candidates, it repeatedly compares the leader against competitors and allocates new execution to unresolved comparisons; the empirical radii do not provide distribution-free coverage or arbitrary-stopping guarantees.Source: §3.4 · p. 5

4 EXPERIMENTAL DESIGN

The experimental design progresses from controlled mechanism tests to finite-data estimation and decision tests, then an independent gate evaluation and external validation after a real model update. The studies use distinct task splits and compare methods under shared data and trajectory budgets where specified.

4 EXPERIMENTAL DESIGN

  • The evaluation hierarchy uses disjoint task splits where required, fixed seeds, shared old-data budgets, and paired evaluation, with the final tool-agent experiment treated as external validation.Full environment construction, estimator definitions, calibration, and statistical tests are provided in appendices.Source: §4 · p. 5

4.1 L1: MECHANISM VALIDATION

  • L1 controls global occupancy-weighted KL while placing updates in shared, branch-differential, or random regions to isolate update location; it also tests natural updates.Its tasks span three semantic families, workflow sizes, and horizons; exact dynamic programming is used only for mechanism tests and evaluation.Source: §4.1 · p. 5

4.2 L2–L3: FINITE-DATA ESTIMATION AND DECISIONS

  • L2–L3 compare reuse, transport, doubly robust estimation, and importance sampling with several refresh rules, evaluating pairwise credit MSE and normalized regret AUC.The confirmation set has 288 tasks, and evaluation uses old-data budgets of 16, 64, or 256 trajectories per action and five seeds.Source: §4.2 · p. 6

4.3 L4 AND REAL-TOOL VALIDATION

  • L4 tests frozen gates on 360 new tasks using regret and new tool steps, while external validation applies a LoRA-updated Qwen3 model in one retail tool environment.The real-tool experiment estimates branch sensitivity from trajectories because exact state visitation is unavailable and sits outside the L1–L4 confirmatory hierarchy.Source: §4.3 · p. 6

5 RESULTS

The results show that credit drift depends on how policy updates align with action-distinguishing branches, while the decision value of transport depends on whether that drift can change the ranking. On an independent test set and after a real model update, DSC-Gate reduces new execution while keeping regret close to a gap-based gate.

5.1 BRANCH SENSITIVITY EXPLAINS CREDIT DRIFT

  • After matching global KL, pairwise sensitivity |G| has an average rank-correlation advantage of 0.839 over global KL for explaining absolute credit drift.Updates concentrated in branch-differential regions produce substantially more pairwise drift than globally similar updates in common regions.Source: §5.1 · p. 6, Figure 3 · p. 7

5.2 TRANSPORT IMPROVES CREDIT ESTIMATES, SELECTIVELY IMPROVING DECISIONS

  • With competing routes, small or moderate updates, and 256 old trajectories per action, transport lowers pairwise credit MSE to 0.001225 from 0.001464 for direct reuse, a 16.3% reduction.The first-order advantage grows with old-data volume but is eroded by the omitted higher-order remainder under larger updates.Source: §5.2 · p. 6, Figure 4 · p. 8
  • In L3, transport with adaptive refresh reaches normalized regret AUC 0.006879, an 18.7% reduction relative to adaptive DR; comparisons against gap-based and occupancy-sensitive refresh have intervals crossing zero.For small and moderate learning updates, true relevance-filtered pairwise reversals are only 0% and 0.116%, so lower credit MSE often does not change the selected action.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 DSC-GATE REDUCES NEW EXECUTION ON THE INDEPENDENT L4 TEST

  • DSC-Gate’s mean terminal regret is 0.00673 versus 0.00669 for Gap-Gate, a +0.00004 difference, while mean new tool steps fall from 472 to 286, a 39.4% reduction.The paired 95% interval for the regret difference is [−0.00010, +0.00018], below the prespecified noninferiority margin of 0.0005.Source: §5.3 · p. 7, Table 2 · p. 8

5.4 A REAL MODEL-WEIGHT UPDATE SHOWS THE SAME CONDITIONAL PATTERN

  • In the real model update, trajectory-level branch sensitivity correlates 0.58 with reference absolute credit drift, versus 0.21 for global KL.With 64 old trajectories per action, transport reduces pairwise MSE by 18.4% relative to direct reuse.Source: §5.4 · pp. 7–8
  • In the real Qwen3-4B LoRA update, DSC-Gate reduces mean new tool steps from 207 to 143, a 30.9% reduction, with mean terminal regret of 0.0291 versus 0.0288 for Gap-Gate.Refresh increases from 15.1% of instances in the lowest branch-sensitivity quartile to 54.7% in the highest.Source: §5.4 · p. 8

6 DISCUSSION

Credit drift and decision failure are distinct: policy changes matter when they affect downstream regions that distinguish actions, and whether drift changes a decision also depends on the action gap. The discussion concludes that selective reuse, correction, and refresh can reduce repeated interaction, within limits on the evidence and setting.

6 DISCUSSION

  • Pairwise branch sensitivity explains why global policy distance can poorly predict credit expiry: updates matter when they overlap states visited differently by candidate root actions.The existing action gap and finite-data uncertainty further determine whether credit needs refreshing.Source: §6 · p. 8
  • First-order transport offers a low-cost correction when policy change is sufficiently local and historical data can estimate it, but improved credit MSE does not always improve regret.The gate uses the action gap to trigger new execution only when drift could plausibly change the decision.Source: §6 · p. 8
  • DSC-Gate uses empirically calibrated radii, and the experiments do not establish distribution-free finite-sample coverage or guarantees under arbitrary adaptive stopping.Reported savings cover target-policy interaction, not historical-data collection, model updates, or calibration; external validation uses one model configuration and one tool environment after a parameter update.Source: §6 · p. 8

7 CONCLUSION

The conclusion argues that historical action credit often remains useful after a policy update, and that selective reuse, correction, and refresh can reduce repeated interaction. The appendices describe the transport derivation, information boundaries, cross-fitting, and ranking-stability rationale behind the approach.

7 CONCLUSION

  • Historical credit can often be reused or corrected after policy updates; on an independent test set, DSC-Gate reduced new tool execution while preserving nearly identical regret.A real tool-agent parameter update showed the same pattern.Source: §7 · p. 9

A.1 NOTATION AND INFORMATION BOUNDARIES

  • L2-L3 estimators receive old trajectories and known behavior- and target-policy tables; dynamic-programming ground truth is reserved for evaluation and error decomposition.Simulator transition tables are not given to trajectory-estimation or allocation interfaces.Source: §A.1 · p. 12

A.2 TRAJECTORY DERIVATION OF FIRST-ORDER TRANSPORT

  • The transport derivation assumes target-policy support under the old policy and an unchanged environment transition mechanism; its first-order remainder contains products of multiple policy changes along a trajectory.The remainder depends on update magnitude and joint occurrence along the same trajectory.Source: §A.2 · p. 12

A.3 BASELINES AND CROSS-FITTING

  • An independently fitted baseline leaves the expected first-order correction unchanged, while cross-fitting prevents each trajectory from fitting its own baseline.The pooled variance remains approximate because the evaluation folds share training information.Source: §A.3 · p. 13

A.4 RANKING STABILITY AND REGRET

  • For nonzero old credit, ranking is preserved when old and target credits have the same sign; with multiple actions, pairwise reversals need not change the optimal choice.The study therefore uses true three-action decision regret rather than pairwise reversal rate as its decision endpoint.Source: §A.4 · p. 13

B.1 L1 MECHANISM EXPERIMENTS

The appendices specify the controlled environments, estimators, calibration, and execution procedures used to study credit transport and DSC-Gate. They separate model updating, credit estimation, target-policy interaction, and independent evaluation.

B.1 L1 MECHANISM EXPERIMENTS

  • L1 matches global update magnitude across differential, common, and random supports, and also tests natural updates; sampled local value changes are a model-assisted diagnostic, not realized execution savings.The workflow environment uses DAGs, reachable-state contexts, and success probabilities sampled from {0.6, 0.8, 1.0}.Source: §B.1 · pp. 13–14

B.2 L2-L3 WORKFLOWS

  • L2-L3 use competing-route workflows with fast, careful, and repair actions; every tool consumes a step, and successful submission returns 1.Serial controls share stage dynamics, but that does not guarantee ranking invariance after a finite-sample update.Source: §B.2 · p. 14

B.3 BASE POLICY AND INDEPENDENT UPDATES

  • Policy updates use separate trajectories from credit-estimation data, while evaluation stores old trajectories in nested budgets of 16, 64, and 256 per task, seed, and route.The base model weights remain fixed in these experiments.Source: §B.3 · p. 14, §C.1 · p. 15

C.1 SHARED DATA AND FOUR ESTIMATORS

  • The estimator comparison includes direct reuse, anchored transport, sequential doubly robust estimation, and weighted importance sampling, with variance floors applied to each estimator.Weighted importance sampling retains its finite-sample bias in evaluation.Source: §C.1 · p. 15

C.2 DEVELOPMENT AND CALIBRATION; C.3 DECISION-SUFFICIENCY RADII

  • Residual variance is estimated on development tasks and frozen before calibration; calibrated radii quantify uncertainty but do not provide formal finite-sample coverage.The radii cover the old gap, correction, and transported gap and are used in the decision rule.Source: §C.2 · pp. 15–16, §C.3 · p. 16

D.1 OLD-DATA PRIORS AND SAMPLING

  • Old-data estimates initialize route priors, and adaptive allocation prioritizes posterior variance relative to the squared action gap.The allocation procedure also has uniform and occupancy-sensitive variants.Source: §D.1 · p. 16

D.2 EXECUTION AND INFORMATION ISOLATION

  • Execution selects routes using currently visible posteriors, counts incomplete trajectories against the budget, and updates posteriors only when trajectories complete.An independent evaluator computes regret and expected success.Source: §D.2 · p. 16

D.3 DECISION-SUFFICIENCY AND BASELINE GATES

  • DSC-Gate resolves comparisons using reuse, transport, or refresh evidence, and restricts new trajectories to routes incident to unresolved comparisons.Execution stops when necessary comparisons are resolved or the budget cap is reached; the stopping rule has no distribution-free coverage guarantee.Source: §D.3 · p. 17

D.4 METRICS

The metrics distinguish decision quality, execution cost, and uncertainty in action selection. L2 evaluates credit estimation and reversal measures; L3 integrates budget curves, while L4 records outcomes at stopping.

D.4 METRICS

  • L2 defines an epsilon-relevant true reversal as cµcπ < 0 with |cπ| > 0.02 and separately records unseen state-time-action tuples and normalized ESS.The reversal statistic concerns true value changes, not incorrect choices caused by estimation noise.Source: §D.4 · p. 17
  • L3 records expected regret, probability of regret above 0.02, expected success, new tool steps, and completed trajectories, then computes normalized AUC over log(1+B).The integrated curve is divided by the total horizontal width log(1537).Source: §D.4 · p. 17
  • L4 records regret, probability of regret above 0.02, actual new tool steps, completed trajectories, refresh and budget-cap incidence, and final resolution proportions.The resolution proportions distinguish reuse, transport, and refresh when each gate stops.Source: §D.4 · p. 18

E STATISTICAL ANALYSIS

The analysis prespecifies confirmation tests and paired evaluation procedures, then reports qualifications on large updates and historical-data diagnostics. L3's inferential result is narrower than its descriptive point estimates.

E STATISTICAL ANALYSIS

  • L1 prespecifies three task-level tests with separate interval-based pass criteria, while L2–L3 use paired resampling and simultaneous intervals for primary comparisons.L1 uses 10,000 task-level bootstrap resamples; L3 resamples tasks within strata and seeds with shared indices.Source: E STATISTICAL ANALYSIS · p. 18
  • Only the transport-versus-DR comparison has a simultaneous L3 interval entirely below zero; the 29.3% and 16.3% gains are descriptive point estimates.The update-slice heat map is also descriptive, not a confirmatory finding.Source: E STATISTICAL ANALYSIS · p. 18
  • The independent L4 test first assesses regret noninferiority to Gap-Gate, then tests whether DSC-Gate uses fewer tool steps than three comparator gates.The primary population equally weights ordinary and branch-selective small or moderate updates across 270 competing-route tasks.Source: E STATISTICAL ANALYSIS · p. 18

F.1 L1 CONFIRMATION TESTS AND LARGE UPDATES

  • At natural-update step size 8, task-averaged first-order MSE exceeds reuse MSE, so the local approximation cannot be extrapolated directly to large updates.This can coexist with positive task-level relative improvement because the summaries weight tasks differently.Source: §F.1 · p. 19

F.2 ALL L2 ESTIMATORS AND NOISE DIAGNOSTICS

  • Historical-data acceptance and post-acceptance ranking error must be read together; unseen-fraction and ESS diagnostics are not formal coverage guarantees.A low post-acceptance error alone could reflect acceptance of only a few cases.Source: §F.2 · p. 20

F.3 ALL L3 METHODS AND UPDATE CONDITIONS

The independent L4 test finds DSC-Gate noninferior to Gap-Gate on regret while using fewer new tool steps. L3 and the quality-crossing analysis specify how to interpret the evaluation measures and budget estimates.

F.3 ALL L3 METHODS AND UPDATE CONDITIONS

  • L3 reports normalized regret AUC for competing routes, while its error-decomposition measure called mean success averages the true values of all three root actions.Mean success is not task success after selecting the optimal root action.Source: Table A5 · p. 20, §F.3 · p. 21

F.4 QUALITY CROSSING AND NEW-EXECUTION BUDGETS

  • Quality crossing is the earliest budget-grid point where both regret targets are met and remain met; the resulting budgets are discrete point estimates, not guaranteed savings rates.An AUC improvement does not establish at least 25% lower execution; 25% was only the original protocol's target effect size.Source: §F.4 · pp. 21–22

F.5 INDEPENDENT L4 GATE TEST

  • In independent L4, regret difference DSC–Gap was +0.00004 and the tool-step ratio was 0.606; ratios versus WIS-Gate and DR-Gate were 0.549 and 0.513.The upper bound of the regret difference was +0.00018, below the 0.0005 noninferiority margin; all three cost-ratio intervals were below 1.Source: §F.5 · pp. 22–23, Table A10 · p. 22
  • Under ordinary updates, DSC-Gate resolves 55.4% of comparisons by reuse and uses 151 new steps on average; under branch-selective updates, 55.9% are refreshed and cost 421 steps.Equal weighting yields a 38.9% refresh rate and 286 new tool steps on average.Source: §F.5 · p. 23, Table A11 · p. 23

G COST ACCOUNTING AND REPRODUCIBILITY MATERIALS

The cost accounting distinguishes online new execution from shared historical and reference-data costs. The reproducibility materials retain detailed metrics and protocol artifacts but omit complete raw trajectories from the lightweight package.

G COST ACCOUNTING AND REPRODUCIBILITY MATERIALS

  • Development and calibration reference trajectories cost 54,656.39 tool steps per confirmation task when amortized once; target-pool infrastructure steps are not an online learner's budget.The 41,824,782 target-pool steps support counterfactual evaluation and are not paid by any one learner.Source: G COST ACCOUNTING AND REPRODUCIBILITY… · p. 23, Table A9 · p. 23
  • The hypothetical amortization count is ceil(15,741,039/s) decisions, assuming old data and reference quantities are reusable and each decision saves s > 0 steps.This is not an overall cost saving demonstrated by the experiment; it depends on transferability and quality-crossing reliability.Source: G COST ACCOUNTING AND REPRODUCIBILITY… · p. 23
  • The lightweight materials include configurations, code, metric tensors, exact values, and derivatives, but omit complete raw trajectories needed for trajectory-level reruns.L4 artifacts preserve its frozen protocol, task-level outcomes, gate states, and bootstrap indices.Source: G COST ACCOUNTING AND REPRODUCIBILITY… · pp. 23–24

Related papers

Loading 2609.29007v1…