Source-linked AI summary
Adapting to Changes in Agent Behavior via Finite-Depth Policy Sensitivity
Lan Shi, Daigo Shishika, Xuan Wang
TL;DR
Adapting a reinforcement-learning policy to another agent’s behavior can require substantial new interaction data, while policy sensitivity depends on second-order derivatives propagated across future interactions. The paper estimates that sensitivity with adjustable finite-depth derivative propagation using reference-environment information. Its bounds tighten with depth, and experiments report lower derivative-estimation errors as depth increases and improved zero-shot return over direct transfer.
Problem
Adapting a policy to changes in another agent’s behavior can require substantial new interaction data, while computing policy sensitivity requires second-order derivatives whose effects propagate across future interactions.
Source: §I · p. 1Method
The method estimates policy sensitivity from reference-environment information by approximating the policy Hessian and mixed derivative with adjustable finite-depth propagation.
Source: §I · p. 2Results
Derivative-estimation errors generally decrease as propagation depth increases, and sensitivity-based initialization improves zero-shot mean return over direct transfer by 0.21-0.66 across six.
Source: A. Sensitivity Estimation Accuracy · p. 6, B. Adaptation Performance · p. 7Takeaways & Limitations
The truncation-error bounds are nonincreasing with propagation depth and vanish at full-horizon propagation, quantifying the depth–accuracy tradeoff.
Source: §I · p. 2, A. Derivative Truncation Error · p. 5The analysis assumes access to the reference transition kernel and its derivative with respect to κ, and treats the boundary value estimate as parameter-independent when truncating value-function derivative propagation.
Source: §VI · p. 7
Abstract
from arXiv · showhide
Adapting a reinforcement learning policy to changes in another agent's behavior typically requires a large amount of new interaction data. Policy sensitivity provides a first-order prediction of how a locally optimal policy changes with a behavioral parameter, but its computation requires second-order derivatives whose effects propagate across future interactions. We develop a finite-depth framework to estimate this sensitivity by approximating the policy Hessian and mixed derivative using information from a reference environment. The method features an adjustable propagation depth which determines where derivative propagation along the trajectory is truncated. We characterize the derivative contributions omitted by finite-depth propagation and derive truncation-error bounds for the approximated derivatives and resulting policy sensitivity. The bounds are nonincreasing with propagation depth and vanish at full-horizon propagation. Using a belief-driven pursuit-evasion game as a validation scenario, the proposed method generally achieves lower derivative-estimation errors as the propagation depth increases and outperforms the baseline methods in both estimation accuracy and policy adaptation. The sensitivity-based initialization improves zero-shot return over direct transfer, and also shows advantages for the subsequent fine-tuning in the target environment.
I. INTRODUCTION
The paper studies adapting a locally optimal policy when another agent’s behavior changes, using reference-environment information to estimate policy sensitivity. It models that behavior change as a parameterized transition kernel and proposes finite-depth derivative propagation with truncation-error guarantees.
I. INTRODUCTION
- A policy optimized for one agent behavior may transfer poorly to another, while retraining from scratch can be costly and full-trajectory estimation or full-horizon propagation can be expensive.The paper motivates sensitivity-based adaptation as a first-order way to adjust a previously learned policy to a nearby environment.Source: §I · p. 1
- Unlike approaches trained across multiple environments, the paper considers adapting a policy learned in one reference environment to a particular change in the interactive agent.The variation is induced by the other agent’s policy rather than directly by physical dynamics or reward,.Source: §I · p. 1
- The contributions are a Bellman-based finite-depth approximation of the policy Hessian and mixed derivative, depth-dependent truncation-error bounds, and validation in a pursuit-evasion game.The bounds are nonincreasing with propagation depth and vanish at full-horizon propagation.Source: §I · p. 2
A. Parameterized Interactive-Agent MDP
- The environment family is indexed by the interactive agent’s behavioral parameter κ; only the agent’s behavior varies, while the initial-state distribution, reward, physical dynamics, and termination rule remain fixed.The induced transition kernel marginalizes over the interactive agent’s action after the learner’s action and intermediate transition.Source: A. Parameterized Interactive-Agent MDP · p. 2
B. Sensitivity-based Adaptation
The paper uses local sensitivity to predict how a locally optimal policy parameter changes when the interactive agent’s behavior changes. This first-order prediction supplies an initialization for adaptation in the target environment.
B. Sensitivity-based Adaptation
- The behavioral change κ_new = κ_old + Δκ defines a target environment where the reference policy may no longer be locally optimal.The policy sensitivity is the derivative of the locally optimal policy parameter with respect to the interactive agent’s behavioral parameter.Source: B. Sensitivity-based Adaptation · p. 3
- Implicit differentiation of the locally optimality condition yields policy sensitivity from the policy Hessian and the mixed derivative with respect to policy and behavior parameters.The Hessian characterizes local objective curvature, while the mixed derivative describes how the policy gradient changes with the behavioral parameter.Source: B. Sensitivity-based Adaptation · p. 3
- For sufficiently small behavioral changes, the sensitivity gives a first-order correction to the reference policy parameter and a natural target-environment initialization.Estimating that sensitivity is equivalent to estimating the Hessian and mixed derivative using reference-environment information.Source: B. Sensitivity-based Adaptation · p. 3
C. Problem of Interest
The paper seeks approximations of the policy Hessian, mixed derivative, and resulting sensitivity, together with bounds on their errors. Its method decomposes derivatives by propagation depth and truncates derivative propagation at a fixed boundary value.
C. Problem of Interest
- The central problem is to construct approximations of the policy Hessian and mixed derivative from reference-environment information and characterize the resulting sensitivity-estimation errors.The sensitivity estimate is computed from the approximated derivatives.Source: C. Problem of Interest · p. 3
- Finite-depth propagation offers a middle ground between trajectory-based derivative estimation, which can require substantial data, and model-based propagation over the full horizon.The approximation combines reference-environment sampling with model-based derivative propagation truncated at adjustable depth T.Source: C. Problem of Interest · p. 3
A. Bellman Derivative Decomposition
- The Bellman decomposition separates derivative contributions within T stages from remaining-horizon boundary terms; the approximation fixes the boundary value and truncates its parameter derivatives.The exact decomposition remains exact for any T; truncation is introduced when the boundary value is replaced and local contributions are recomputed through Bellman recursion.Source: §III · p. 3, A. Bellman Derivative Decomposition · p. 4
- The method represents conditional returns and state distributions through Bellman and forward state-distribution recursions to compute the local derivative contributions.The recursions assume differentiation can be interchanged with the relevant integrals under standard regularity conditions.Source: A. Bellman Derivative Decomposition · pp. 3–4
B. Finite-Depth Sensitivity Approximation
The finite-depth approximation samples initial states from the reference environment and uses an adjustable depth to limit derivative propagation. It holds the boundary-value estimate fixed with respect to the parameters, then uses the estimated sensitivity to initialize target-environment adaptation.
B. Finite-Depth Sensitivity Approximation
- The procedure samples N initial states from the reference distribution and selects a propagation depth T, with the boundary value estimated by a reference-policy critic or fitted from reference trajectories.The boundary estimate is held fixed with respect to θ and κ during derivative propagation.Source: B. Finite-Depth Sensitivity… · p. 4
- At T = H, the boundary value is zero, so the construction retains full-horizon derivative propagation without truncation.At shorter depths, the boundary estimate retains future-return information while its parameter derivatives are set to zero.Source: B. Finite-Depth Sensitivity… · p. 4
- The resulting sensitivity estimate supplies the initial policy parameter for adaptation in the target environment.The truncated recursions propagate state distributions and their derivatives forward from sampled initial states, then use sample averages for the approximation.Source: B. Finite-Depth Sensitivity… · p. 4
IV. SENSITIVITY ERROR ANALYSIS
The analysis isolates structural truncation error from finite-sample and value-function approximation errors. It uses an ideal setting to make that separation, while examining the practical effects of the other errors empirically.
IV. SENSITIVITY ERROR ANALYSIS
- The analysis isolates finite-depth structural truncation from finite-sample and value-function approximation errors.These two error sources are treated as complementary to the truncation error.Source: §IV · p. 4
- In the ideal setting, the boundary value and sample averages are replaced by exact values and expectations under ρ0.Source: §IV · p. 4
- The practical effects of finite-sample and value-function approximation errors are examined empirically in Section V.Source: §IV · p. 4
A. Derivative Truncation Error
Finite-depth propagation omits parameter-derivative contributions from stages at or beyond the cutoff, and the paper bounds the resulting derivative and sensitivity errors. Under the stated conditions, these bounds do not increase with propagation depth and vanish at the full horizon.
A. Derivative Truncation Error
- Uniform bounds on reward, policy derivatives, and transition-kernel derivatives bound omitted terms; undifferentiated probability factors do not enlarge the bounds.The transition derivative is bounded in total variation.Source: A. Derivative Truncation Error · pp. 4–5
- Finite-depth recursion retains parameter derivatives through stages 0 to T − 1 and omits terms involving policy or transition factors at stage T or later.The omitted contributions arise because the boundary value is held fixed with respect to θ and κ.Source: A. Derivative Truncation Error · p. 5
- For rewards at stages q ≥ T, the Hessian estimate retains T single-policy second derivatives and T(T − 1) cross-stage terms, while the mixed-derivative estimate retains T² policy-transition terms.The full counts are q + 1, q(q + 1), and (q + 1)², respectively.Source: A. Derivative Truncation Error · p. 5
- The derivative-error bounds are nonincreasing in T and vanish at T = H; the sensitivity bound follows when η_H(T) < m.Here m is the minimum eigenvalue of −H_old, which is positive under Assumption 1.Source: A. Derivative Truncation Error · pp. 5–6
- Increasing T removes omitted terms and reduces their counts at remaining stages, making both derivative-error bounds nonincreasing and empty at T = H.Source: A. Derivative Truncation Error · p. 5
V. SIMULATION
The simulations use a pursuit–evasion game in which the defender’s behavior parameter changes the environment experienced by the learning agent. Two grid scales serve derivative-accuracy and policy-adaptation evaluations.
V. SIMULATION
- The game keeps physical dynamics and task fixed while the defender acts as the interactive agent and the learning agent controls two attackers.Attackers move before the defender, whose policy affects the next state.Source: §V · p. 6
- The defender’s κ mixes belief-blind and belief-aware policies, controlling its reliance on a belief about which attacker holds the hidden key.The belief-aware policy mixture is ψ(· | ˜s_t; κ) = (1 − κ)ψ0 + κψ1, with κ in [0, 1].Source: §V · p. 6
- The 6×6, H = 15, γ = 0.9 game benchmarks derivative and sensitivity accuracy, while the 20×20, H = 60, γ = 0.99 game evaluates zero-shot return and adaptation.PPO trains the learning agent’s policy; Fig. 2 plots errors of H, M, and S against propagation depth T.Source: §V · p. 6, Figure 2 · p. 6
A. Sensitivity Estimation Accuracy
The proposed estimator’s error falls as propagation depth increases, and at depths 5 and 7 it outperforms the trajectory likelihood-ratio estimator. Its use of initial-state samples and conditional transition-kernel expectations reduces sampling of subsequent actions and transitions.
A. Sensitivity Estimation Accuracy
- With critic estimates and N=106 sampled initial states, errors decrease initially with T and then approach a plateau near 10^-3.At T=H, truncation error is zero; remaining error comes from sampling or numerical computation.Source: A. Sensitivity Estimation Accuracy · p. 6
- Under exact boundary values and exact expectations, errors decrease with T, consistent with retaining more derivative terms.The sensitivity error also decreases as the derivative bounds tighten.Source: A. Sensitivity Estimation Accuracy · p. 6
- Increasing T lowers the proposed estimator’s error curve, while conditional transition-kernel expectations avoid sampling subsequent actions and transitions.The proposed method samples initial states, whereas the likelihood-ratio estimator uses complete trajectories.Source: A. Sensitivity Estimation Accuracy · p. 6
- At T=5 and 7, the proposed method consistently achieves lower error than the trajectory likelihood-ratio estimator; at T=3, the latter eventually becomes more accurate as N grows.Fig. 3 compares relative estimation error versus N; its columns correspond to T=3, 5, and 7.Source: Figure 3 · p. 7, A. Sensitivity Estimation Accuracy · p. 7
B. Adaptation Performance
Across six transfers, sensitivity-based initialization improves zero-shot return over direct transfer and can retain an advantage during target-environment fine-tuning. Both transfer methods outperform training from scratch within the fixed training budget.
B. Adaptation Performance
- The transfer methods use the same reference policy and fine-tuning settings; their only difference is policy initialization.Sensitivity is estimated with T=1 and N=6000 per estimate, and training uses PPO for 3×10^5 steps with ten seeds.Source: B. Adaptation Performance · p. 7
- Sensitivity-based initialization improves zero-shot mean return over direct transfer by 0.21-0.66 across the six transfers.The advantage persists during fine-tuning in several cases and gradually narrows in others as both methods adapt.Source: B. Adaptation Performance · p. 7
- Both transfer methods perform better than training from scratch within the fixed budget.The comparison covers six changes in κ, including increases and decreases in the defender’s behavioral parameter.Source: B. Adaptation Performance · p. 7
VI. CONCLUSION
The paper presents a finite-depth framework for policy-sensitivity estimation and proves that its truncation-error bounds decrease with propagation depth. Pursuit–evasion experiments validate the approach, while the analysis assumes access to the reference transition kernel and its behavioral-parameter derivative.
VI. CONCLUSION
- Fig. 4 reports mean discounted return over target-environment training steps for six changes in κ, with insets expanding the transfer-method comparison.The caption describes the figure’s axes and scope, not the outcome of the comparison.Source: Figure 4 · p. 7
- The truncation-error bounds decrease with propagation depth, and the paper validates the finite-depth framework in pursuit–evasion experiments.The adjustable depth trades off model-based derivative propagation and approximation accuracy.Source: §VI · p. 7
- The analysis assumes access to the reference transition kernel and its derivative with respect to κ.The method also treats the boundary value estimate as parameter-independent when truncating value-function derivative propagation.Source: §VI · p. 7
APPENDIX
The appendix derives derivative recursions by differentiating the Bellman and boundary expressions. It contrasts exact full-horizon recursions with truncated recursions initialized at depth T.
Proof of Lemma 1
- Differentiating the boundary integral yields separate terms from the state distribution and the value gradient.The interchange of differentiation and integration relies on the regularity condition stated in Section III-A.Source: Proof of Lemma 1 · p. 7
- The appendix gives derivative recursions for fixed initial states and obtains them by differentiating the Bellman recursions.The recursions supply the derivatives used in the decomposition.Source: Proof of Lemma 1 · p. 8
- Exact recursions start at stage H, while truncated estimates start at stage T with bVT and zero parameter derivatives of the boundary value.The exact backward recursions provide values and derivatives at stage T and earlier.Source: Proof of Lemma 1 · p. 8