Source-linked AI summary
A Bellman Optimality Equation for Plasticity
Jeremy Lucas, Doina Precup
TL;DR
Continual reinforcement learning lacks direct optimization methods for plasticity under its generalized directed-information definition, despite extensive work on empowerment. This paper derives a Bellman optimality equation for plasticity, proves contraction of its operator, and evaluates it in a Control-Gated MDP, where the learned policy reaches approximately 0.92 bits of plasticity and 0 bits of empowerment.
Problem
Direct optimization of information-theoretic plasticity remains unexplored, although empowerment optimization has established methods and both quantities are proposed as important for continual reinforcement learning.
Method
The paper derives a Bellman optimality equation for plasticity, proves its Bellman operator is a contraction, and uses history-dependent tabular optimization with a Multi-Weight Knapsack reduction.
Results
The learned policy converges to approximately 0.92 bits of plasticity and 0 bits of empowerment in the Control-Gated MDP.
Takeaways & Limitations
The paper provides preliminary evidence that plasticity can be optimized in Markov decision processes through a Bellman formulation analogous to empowerment optimization.
Takeaways & Limitations
The current approach is limited to tabular settings, with online sample-based scaling to continuous, high-dimensional environments left for future work.
Abstract
from arXiv · showhide
In continual reinforcement learning, carefully managing the stability-plasticity tradeoff remains a core challenge. Recent work by Abel et al. (2025) formalized this dilemma by defining plasticity as the generalized directed information from an agent's observations to its actions, and empowerment as the generalized directed information from its actions to its observations. This formulation successfully reframes the traditional stability-plasticity tradeoff as an empowerment-plasticity tradeoff. However, while extensive literature exists on optimizing for empowerment, there is currently no research addressing the optimization of plasticity under this new definition. This paper presents preliminary work toward optimizing plasticity within Markov decision processes. We show that there exists a Bellman optimality equation for optimizing plasticity similar to previous work for empowerment.
1 Introduction
The paper frames continual reinforcement learning as an empowerment-plasticity tradeoff and addresses the lack of direct optimization methods for information-theoretic plasticity. It proposes deriving a Bellman optimality equation for plasticity within Markov decision processes.
- Plasticity is defined as information flowing from observations to actions, whereas empowerment measures information flowing from actions to subsequent observations.
- Both empowerment and plasticity must be actively optimized to support robust continual learning agents under the proposed formulation.
- Existing work has developed multiple empowerment-maximization methods, including Bellman and deep variational formulations.
- Direct optimization of plasticity under its new information-theoretic definition remains unexplored, motivating this paper’s Bellman formulation.
2 Background
The background develops directed-information definitions of empowerment and plasticity and positions the paper’s objective as making plasticity optimization tractable in Markov decision processes. The paper isolates the information-theoretic plasticity objective by setting external rewards aside.
- Empowerment is defined as directed information from an agent’s actions to its subsequent observations, extending earlier channel-capacity formulations.
- Abel et al. define empowerment and plasticity as generalized directed-information quantities over action and observation windows.
- This paper studies a single agent interacting with a specific environment rather than the broader agent-environment sets in the definitions.
- Prior value-based methods make empowerment optimization tractable through generalized Bellman equations, providing the model for the plasticity formulation.
- The paper sets external rewards to zero to isolate the information-theoretic value of plasticity and its resulting policies.
3 The Bellman Optimality Equation for Plasticity
The paper constructs a Bellman optimality equation for plasticity by using a directed-information step reward and a history-dependent future value. It establishes contraction-based foundations and derives a dynamic-programming optimization procedure for tabular settings.
- The plasticity formulation uses I(S_t+1, A_t+1|S_t, A_t) as the step reward and V(S_t+1, A_t+1) as the future value.
- The associated Bellman operator is proven to be a contraction, establishing theoretical foundations for the plasticity optimality equation.
- For tabular learning, the policy must include historical state-action context, and action distributions across reachable next states must be optimized jointly.
- The optimization can be restricted to deterministic policies, but naïve enumeration requires searching |A||S| action-distribution combinations for each history context.
- Because shared q marginals couple next-state action choices, the paper reduces the joint search through a Multi-Weight Knapsack dynamic program.
- Tabular value iteration initializes values and policies, repeatedly solves the knapsack dynamic program, and outputs an optimal value function and policy.
4 The Plasticity-Empowerment Landscape
The paper introduces a minimal Control-Gated MDP to isolate interactions between plasticity and empowerment and maps its empirical frontier by enumerating stochastic policies. The benchmark contains two states and two actions with asymmetric controllability, producing distinct policy patterns for the two metrics.
- The Control-Gated MDP is the simplest identified environment that retains a rich, non-trivial plasticity-empowerment landscape.
- The benchmark contains two states and two actions, enabling exact numerical optimization and complete policy enumeration.
- Its asymmetric controllability combines a highly empowering state S1 with a nonempowering state S0.
- A grid search over stochastic policies π(a | s) at granularity 0.01 maps the discretized empirical plasticity-empowerment rate landscape.
- Plasticity is bounded by log2 |A| bits, whereas empowerment is bounded by log2 |S| bits.
- Highly plastic policies occupy blue regions, while empowering policies show a mixture of blue and red regions in the policy landscape.
5 Experiments
Experiments evaluate the plasticity Bellman optimality equation in a control-gated benchmark and two scaled environments. Value iteration converges in the benchmark, while the learned policies favor environments and histories associated with higher plasticity.
- 5.1 Control-Gated MDP: Value iteration is monotonically non-decreasing and converges to the optimal value function across all history contexts in the Control-Gated MDP.
- 5.1 Control-Gated MDP: Deterministic transitions (1, 0) and (1, 1) yield the lowest values, with (1, 1) minimizing value because it returns to state 1.
- 5.1 Control-Gated MDP: The Control-Gated MDP policy converges to approximately 0.92 bits of plasticity and 0 bits of empowerment.Early variation reflects random selection among multiple equally valued optimal policies with different total plasticity.
- 5.2 Road Environment: In the Road Environment, attempting to leave the road from histories (0, 0) and (4, 2) yields the least value.The environment has five lane states, three turning actions, and uniformly sampled wind bounded within the road.
- 5.3 Two-Room Environment: In the Two-Room Environment, policy simulation navigates toward the bridge to the noisy room to capture the maximum plastic value.The gridworld combines a fully noisy room with a fully deterministic room and contains 35 states.
6 Future Work
Future work proposes scaling the plasticity-optimization framework to continuous, high-dimensional environments and combining it with empowerment-maximization methods.
- Deep neural networks could parameterize the policy and value function in an online, sample-based algorithm for continuous, high-dimensional environments.The passage presents this as a scaling direction rather than an implemented result.
- Integrating plasticity optimization with empowerment-maximization algorithms could support agents that balance plasticity, empowerment, and external reward during continual learning.
- The framework operates at a fundamental algorithmic level without immediate negative societal impacts or direct applications to sensitive domains.
A.1 Contraction Mapping Property of the Bellman Operator
The Bellman operator is defined on bounded value functions over history state-action contexts and shown to be a γ-contraction, yielding a unique fixed point.
- The operator T maps bounded, real-valued functions on history state-action space to the same function space.The value-function space uses the supremum norm ∥V∥∞ = sup_s,a |V(s, a)|.
- The proof compares arbitrary value functions pointwise, uses the maximum inequality, cancels identical logarithmic terms, and bounds the remaining value difference by the supremum norm.
- For γ ∈ (0, 1), the Bellman operator T is a γ-contraction under the supremum norm and therefore has a unique fixed point V ∗.The contraction result is established uniformly across state-action contexts.
A.2 Limiting the Bellman Operator to Policy Endpoints
The Bellman optimization objective is convex in the policy, so its maximum can be attained at deterministic policy endpoints rather than requiring stochastic policies.
- Restricting the policy search to deterministic endpoints preserves the Bellman operator’s value for any value function V.
- The objective function inside the Bellman operator is convex in π, and its maximum over the compact policy space is attained at an extreme point corresponding to deterministic action selection.
- The negative-entropy, cross-entropy baseline, and value-expectation terms establish convexity because the first is strictly convex while the latter two are affine.
- Bauer’s Maximum Principle implies that optimizing the convex objective over the probability simplex reaches an extreme point with all probability mass assigned to one action.
B Experimental Details
Table 1 reports hyperparameter configurations for the Control-Gated MDP, Road Environment, and Two-Room Environment.
- Table 1 organizes hyperparameter configurations for the Control-Gated MDP, Road Environment, and Two-Room Environment.