Source-linked AI summary
Policy Complexity, Reaction Time, and Bounded Rationality in Reinforcement Learning
James Wu, Chris R. Sims
TL;DR
Biological learning is constrained by limits on the state information available to guide behavior, a cost standard reinforcement learning typically omits. MI-SARSA adds mutual-information regularization to on-policy SARSA and uses the same state-wise cost to predict reaction time. Its results show a reward-complexity tradeoff and a robustness-capacity tradeoff under environment shift.
Problem
Biological agents face limits on perception, attention, and working memory, while standard reinforcement learning typically omits the internal costs of state-dependent policies.
Source: §1 · p. 1Method
Results
MI-SARSA exhibits a reward-complexity tradeoff, links state-wise KL cost to reaction time, and shows that greater regularization smooths adaptation under environment shift while reducing asymptotic performance.
Source: §4.3 · p. 4, §4.4 · p. 5, §5 · p. 7Takeaways & Limitations
State information is used when its expected benefit justifies computational cost and increased behavioral latency, linking reward maximization, policy compression, and reaction time.
Source: §5 · p. 7The model is evaluated in simple, fully observable tabular tasks and is not fit to human data, so it is a framework rather than a detailed descriptive account.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Biological agents do not learn under conditions of unlimited computation. For humans, learning and choice are shaped by constraints on perception, attention, and working memory, which limit how much state information guides behavior and therefore bound policy complexity. Standard reinforcement learning models typically optimize reward without explicitly representing these internal costs, making them less suitable as models of biological intelligence. We derive MI-SARSA, an on-policy temporal-difference algorithm that incorporates mutual-information regularization through a learned marginal action prior and a penalty on state-specific deviations from that prior. This yields a sequential learning model in which state information is used selectively when its expected return benefit justifies the added informational cost. Critically, the same state-specific information cost that governs policy compression also generates trial-level predictions for reaction time, distinguishing MI-SARSA from most reinforcement learning models, which predict choices or returns but not latency. Empirically, MI-SARSA produces a reward-complexity tradeoff, and stronger information penalties produce simpler policies with lower control costs and faster reaction times. Under environment shift, increasing regularization reduces post-switch performance degradation but also lowers asymptotic return, revealing a robustness-capacity tradeoff. Together, these results position MI-SARSA as a model of bounded sequential learning under cognitive constraints.
1 Introduction
Biological learners face limits on the state information they can use, while standard reinforcement learning typically omits the internal costs of state-dependent behavior. MI-SARSA applies an information penalty to sequential learning and links the cost of state-specific choices to reaction time.
1 Introduction
- Constraints on perception, attention, and working memory limit state information use and bound the complexity of policies biological agents can implement.Standard reinforcement learning typically optimizes reward without explicitly accounting for the internal costs of representing and deploying state-dependent behavior.Source: §1 · p. 1
- Mutual information between states and actions measures state dependence: lower values correspond to simpler behavior, while higher values reflect specialized, state-contingent control.The framework describes state information as useful when it improves reward, but only to the extent justified by its informational cost.Source: §1 · p. 1
- MI-SARSA penalizes state-specific policy deviations from an adaptive action prior, equivalent to regularizing mutual information, so state information is used when reward justifies its cost.The penalty is a Kullback-Leibler divergence between the state-conditional policy and marginal action prior.Source: §1 · pp. 1–2
- The model extends information-theoretic constraints to on-policy SARSA for sequential decision making under internal resource constraints.Earlier work focused on control performance and exploration, actor-critic formulations [6], or one-shot decision tasks.Source: §1 · p. 2
- The same state-specific information cost links policy complexity to reaction time, predicting that more specialized choices take longer.The formulation aims to explain both which actions are selected and why choices requiring greater policy differentiation may take longer.Source: §1 · p. 2
2 Related Work
The paper situates MI-SARSA within established information-theoretic approaches to reinforcement learning and bounded rationality. Its contribution is to use a policy-complexity penalty in sequential temporal-difference learning and interpret the state-wise cost as a predictor of reaction time.
Info-theoretic Regularization in RL
Policy Compression and Bounded Rationality in Cognitive Science
- Cognitive-science work frames policy complexity as mutual information between states and actions and connects bounded action selection with response time and information-processing cost.Related work also connects efficient coding to human generalization and cognitive effort to information needed to update internal representations.Source: §2 · p. 2
Present Contribution
- The contribution is a new use of an established information-theoretic objective: a policy-complexity penalty embedded in temporal-difference learning and interpreted as a reaction-time predictor.The resulting framework links reward, reaction time, and bounded choice within sequential reinforcement learning.Source: §2 · p. 2
3 Method
MI-SARSA adds a mutual-information policy-complexity cost to fully observable, on-policy SARSA. An adaptive marginal action prior supplies the reference for that cost, which enters learning through a regularized immediate reward.
3.1 Problem Setting and Objective
- The control objective trades expected return against mutual information between states and actions, with β controlling the reward-versus-compression tradeoff.At β = 0 the objective reduces to standard reward maximization; larger β increasingly penalizes state-specific action policies.Source: §3.1 · p. 3
3.2 Adaptive Action Prior and Policy-Complexity Penalty
- The marginal action prior tracks long-run action frequencies independently of state and serves as the reference distribution for the mutual-information penalty.It starts uniform and is updated online using an exponential moving average of observed actions.Source: §3.2 · p. 3
- MI-SARSA uses the expected KL divergence between the state-conditional policy and marginal prior as its state-specific information cost.The cost is incorporated into immediate reward, penalizing prior deviations while allowing them when expected reward gain justifies the informational cost.Source: §3.2 · p. 3
3.3 MI-SARSA
- MI-SARSA applies the regularized immediate reward within an on-policy SARSA update, sampling the next action from the current policy.Compared with standard SARSA, the immediate reward is replaced by its mutual-information-regularized counterpart.Source: §3.3 · p. 3
4 Results
Across two navigation environments, MI-SARSA exhibits tradeoffs among reward, policy complexity, reaction time, and robustness to environment shifts. Stronger regularization produces simpler policies and smoother transitions, but reduces asymptotic return.
4.1 Environments
- The study evaluates MI-SARSA in Frozen Lake and Cliff Walking, grid domains with hazards and regions requiring precise control.The tasks use distinct hazard and reward structures, with state represented by grid location and four primitive movement actions.Source: §4.1 · p. 4
4.2 Evaluation Protocol
- The main reward-complexity analysis sweeps regularization strength, trains policies for 1,000 episodes, and evaluates them for 100 episodes across 50 seeds.Return is mean cumulative reward; policy complexity is estimated from log π(a | s) − log p(a) over visited state-action pairs.Source: §4.2 · p. 4
4.3 Reward-Complexity Tradeoff
- Across both environments, expected return rises with policy complexity, while lower-complexity policies show greater variability across seeds.Higher complexity reflects greater use of state-specific information; more structured policies tend to produce more stable outcomes.Source: §4.3 · pp. 4–5, Figure 1 · p. 5
4.4 Policy Complexity and Reaction Time
- Stronger regularization reduces policy complexity and reaction time, while behaviorally consequential states retain slower decisions.Reaction time is linked to the state-specific information cost of constructing an action distribution from the marginal prior.Source: §4.4 · p. 5, Figure 2 · p. 6
4.5 Adaptation Under Environment Shift
- Stronger regularization reduces post-switch performance degradation but lowers final per-environment return in both navigation domains.The results characterize a reward-robustness tradeoff rather than a single optimal regularization level.Source: Figure 3 · p. 6, §4.5 · p. 7
- For environment-shift tests, agents train across sequences of 20 related environments, with each fixed environment used for a 100-episode block.The study measures the immediate transition in performance and averages final performance over each block’s last 20 episodes.Source: §4.5 · pp. 6–7
5 Discussion
MI-SARSA links reward, policy complexity, and reaction time through information costs, while moderate constraints can improve adaptation under environment shift. The findings are a framework-level contribution, evaluated only in simple, fully observable tabular tasks and not fitted to human data.
5 Discussion
- Higher return is associated with more state-specific policies, while stronger regularization compresses behavior toward a simpler action prior.Source: §5 · p. 7
- The state-wise Kullback-Leibler cost governing policy compression also predicts which states should produce longer decision times.Source: §5 · p. 7
- Moderate policy-complexity constraints can improve adaptation under environment shift by reducing over-specialization.Source: §5 · p. 7
- The model was evaluated in simple, fully observable tabular tasks and was not fit to human data, so it is presented as a framework rather than a detailed descriptive account.Source: §5 · p. 7