PaperlayerView exact revision on arXiv

Source-linked AI summary

Emotion-Inspired Learning Signals (EILS): A Homeostatic Framework for Adaptive Autonomous Agents

Dhruv Tiwari

arXiv:2512.22200v1cs.LG

TL;DR

The paper addresses the fragility of agents trained with static external rewards under sparse feedback and distribution shifts. It proposes EILS, an Internal State Module that converts interaction-history signals into homeostatic modulation of optimization, and reports improved adaptation with low computational overhead, while acknowledging additional sensitivity hyperparameters.

  • Problem

    Static reward-based agents lack internal regulatory autonomy for sparse-feedback exploration, distribution-shift adaptation, and runtime plasticity.

  • Method

    EILS adds an Internal State Module that derives Curiosity, Stress, and Confidence from interaction history and modulates the optimizer online.

  • Results

    EILS agents recovered to 90% performance within 50 episodes after a Dynamic CartPole collapse, while computational overhead remained below 5% versus vanilla PPO.

  • Takeaways & Limitations

    Internal Stress enabled rapid plasticity after distribution shifts, while homeostatic Curiosity supported broad state coverage without ensemble-method computational cost.

  • Takeaways & Limitations

    EILS introduces sensitivity hyperparameters that require tuning, and excessive Stress sensitivity may mistake stochastic noise for distribution shift.

Abstract

from arXiv · show

The ruling method in modern Artificial Intelligence spanning from Deep Reinforcement Learning (DRL) to Large Language Models (LLMs) relies on a surge of static, externally defined reward functions. While this "extrinsic maximization" approach has rendered superhuman performance in closed, stationary fields, it produces agents that are fragile in open-ended, real-world environments. Standard agents lack internal autonomy: they struggle to explore without dense feedback, fail to adapt to distribution shifts (non-stationarity), and require extensive manual tuning of static hyperparameters. This paper proposes that the unaddressed factor in robust autonomy is a functional analog to biological emotion, serving as a high-level homeostatic control mechanism. We introduce Emotion-Inspired Learning Signals (EILS), a unified framework that replaces scattered optimization heuristics with a coherent, bio-inspired internal feedback engine. Unlike traditional methods that treat emotions as semantic labels, EILS models them as continuous, homeostatic appraisal signals such as Curiosity, Stress, and Confidence. We formalize these signals as vector-valued internal states derived from interaction history. These states dynamically modulate the agent's optimization landscape in real time: curiosity regulates entropy to prevent mode collapse, stress modulates plasticity to overcome inactivity, and confidence adapts trust regions to stabilize convergence. We hypothesize that this closed-loop homeostatic regulation can enable EILS agents to outperform standard baselines in terms of sample efficiency and non-stationary adaptation.

I. INTRODUCTION

EILS frames robust autonomy as a problem of regulatory autonomy rather than representational capacity. It introduces biologically inspired internal signals to regulate learning under sparse feedback, distribution shifts, and changing task demands.

  • Static extrinsic rewards make standard agents brittle when feedback is sparse, delayed, or vulnerable to reward hacking.
  • Distribution shifts expose the limits of fixed-reward optimization because behavior that is efficient today may become harmful tomorrow.
  • The Stability-Plasticity Dilemma requires agents to learn new patterns while retaining existing knowledge, which fixed learning-rate schedules cannot reliably balance.
  • EILS adds an Internal State Module that computes Curiosity, Stress, and Confidence from surprise and success histories to modulate the optimizer online.
  • The framework positions emotion-inspired regulation as a computational bridge between biological homeostasis and adaptive reinforcement learning.
  • Stress models a tonic norepinephrine-like mode that increases learning rate and plasticity when prediction errors accumulate.

2) The Anterior Cingulate Cortex (Confidence):

EILS uses Confidence as a stability signal that suppresses unnecessary exploration when value predictions are consistent. Its Curiosity mechanism instead bounds novelty-seeking with a homeostatic setpoint, while Stress and self-tuning provide complementary adaptation mechanisms.

  • 2) The Anterior Cingulate Cortex (Confidence):: Confidence estimates environmental stability from value-function variance and inhibits stochastic exploration when predictions are reliable.
  • 2) The Anterior Cingulate Cortex (Confidence):: High Confidence supports low-energy exploitation by reducing cognitive overhead once outcomes become predictable and positive.
  • EILS bounds Curiosity with a homeostatic setpoint to avoid the Noisy-TV failure mode associated with treating all novelty as positive.
  • Stress dynamically changes plasticity during task switching, unlike rigid consolidation methods that constrain updates uniformly.
  • EILS tunes itself from internal state rather than requiring a population of agents for evolutionary hyperparameter optimization.

4) Autonomous Agents and LLMs:

EILS is presented as a physiological regulation layer for autonomous agents, complementing semantic reasoning systems with internal control of learning. It extends the MDP with recurrent emotional state derived from interaction history.

  • 4) Autonomous Agents and LLMs:: LLM-based agents such as Voyager and Reflexion use prompting or verbal reinforcement, whereas EILS regulates the learning process at a lower physiological level.
  • EILS extends the standard MDP with a recurrent internal-state vector and an augmented state space S × S_int.
  • The internal state is computed from a sliding-window history of prediction errors and rewards.
  • Stress, Curiosity, and Confidence represent failure, information need, and value-prediction stability, respectively.
  • The homeostatic objective penalizes deviation from an optimal epistemic setpoint, treating excessive Stress as increased discomfort.

IV. METHODOLOGY: THE EILS ARCHITECTURE

EILS wraps a standard Actor-Critic learner with an Internal State Module that estimates epistemic conditions using exponentially filtered prediction signals. Stress responds asymmetrically to surprising failures and controls adaptive plasticity.

  • IV. METHODOLOGY: THE EILS ARCHITECTURE: EILS operates as an algorithm-agnostic meta-regulatory wrapper around Actor-Critic methods, with PPO serving as the implementation’s policy learner.
  • IV. METHODOLOGY: THE EILS ARCHITECTURE: The Internal State Module uses Exponential Moving Averages to estimate epistemic status while filtering stochastic environmental noise.
  • 1) Stress (σt): The Frustration & Plasticity Signal:: Stress accumulates unpleasant prediction-error shocks and functions as a plasticity signal.
  • 1) Stress (σt): The Frustration & Plasticity Signal:: Stress is updated through an EMA with decay rate η_σ, which controls the persistence of the stress response.
  • 1) Stress (σt): The Frustration & Plasticity Signal:: ReLU rectification makes Stress sensitive to surprising failures while preventing positive prediction errors from increasing it.

2) Curiosity (κt): The Entropy Signal:

Curiosity is computed from forward-model prediction error and acts as an information-gain proxy, rising in novel regions and fading in familiar ones. The resulting internal states modulate PPO hyperparameters through homeostatic transfer functions.

  • Curiosity signal: Curiosity derives from a self-supervised Forward Dynamics Model that predicts the next state from the current state and action.The raw curiosity impulse is based on the model’s transition-prediction error.
  • Curiosity signal: The curiosity impulse is defined by the squared Euclidean norm of the dynamics-model prediction error.
  • Curiosity signal: Prediction error approaches zero in familiar state-space regions but spikes in novel regions, making curiosity a proxy for information gain.Unlike count-based exploration, the mechanism is intended for high-dimensional continuous state spaces.
  • Homeostatic modulation: The computed internal states dynamically modulate PPO hyperparameters through transfer functions designed to reduce the Homeostatic Deficit.
  • Homeostatic modulation: Stress scales the learning rate to address the Frozen Synapse problem.
  • Homeostatic modulation: A tanh-based bound caps stress-driven learning-rate modulation, limiting exploding gradients while permitting rapid unlearning of obsolete behaviors.The cap is specified as (1 + λσ) × αbase.

2) Adaptive Entropy Coefficient (βt):

The adaptive entropy coefficient increases exploration when curiosity falls below its setpoint, while confidence narrows PPO updates during stable mastery. Together, these controls implement homeostatic regulation of exploration and policy stability.

  • Adaptive entropy coefficient: The entropy coefficient follows homeostatic setpoint logic to govern exploration.
  • Adaptive entropy coefficient: When curiosity is below its setpoint, the sigmoid increases βt and raises policy entropy, forcing random actions to discover new dynamics.This inverted intrinsic-motivation logic treats low curiosity as boredom.
  • Adaptive clip range: The PPO clip range scales with Confidence to stabilize learning during mastery.
  • Adaptive clip range: When Confidence is high, the trust region is diminished so updates remain conservative and reduce catastrophic forgetting of the optimal policy.The clip range ε defines permissible divergence between old and new policies.
  • Architecture and overhead: The shared Actor-Critic backbone uses a 2-layer, 64-unit Tanh MLP, while the Forward Dynamics Model is separate.The model takes concatenated state-action pairs as input.
  • Architecture and overhead: EILS adds less than 5% wall-clock time overhead over vanilla PPO, with scalar internal-state calculations and a small Forward Model pass.The internal-state calculation is O(1), while the Forward Model adds O(N 2) computation.
  • Architecture and overhead: Confidence calculation stores H = 50 scalars in an insignificant history buffer.

V. EXPERIMENTAL SETUP

The experiments test whether EILS improves adaptability and sample efficiency across environments targeting non-stationarity, sparse rewards, and rule reversal. The EILS-PPO loop updates internal states from trajectory errors before adapting PPO parameters.

  • Experimental rationale: Three diverse environments target standard Deep RL failure modes: non-stationarity, sparse rewards, and rule reversal.
  • Training loop: EILS-PPO initializes policy, value, and forward models alongside a three-component internal state and adaptive-optimization parameters.
  • Training loop: For each trajectory step, the loop computes TD and dynamics errors, updates Stress and Curiosity, estimates Confidence, and modulates α, β, and ε.
  • Training loop: The PPO update maximizes the clipped objective using adaptive parameters, while the Forward Model is updated by minimizing dynamics error.
  • Non-stationarity: CartPole: The CartPole experiment introduces a catastrophic distribution shift after standard physics training to test plasticity.Standard CartPole is often solved within 200 episodes.
  • Non-stationarity: CartPole: After episode 500, gravity rises to 30.0 and pole mass doubles to mp = 0.2, making the previously learned policy obsolete.
  • Non-stationarity: CartPole: Recovery time is the number of episodes needed to return to a moving-average reward of ≥195 after the shift.
  • Sparse exploration: GridWorld: The 20×20 GridWorld contains one goal in a discrete state space of S ∈R400 while the agent monitors raw coordinates.

2) The “Sparse Maze” (Exploration):

The sparse-maze and rule-reversal tasks evaluate exploration and cognitive flexibility under limited or changing feedback. Their setup contrasts sparse goal discovery with a reward reversal that can induce perseveration when entropy collapses.

  • Sparse Maze: The GridWorld reward is extremely sparse, providing reward only under the specified sparse-reward structure.
  • Sparse Maze: Sample efficiency is measured by the number of steps required to determine the goal for the first time.This tests whether Curiosity drives coverage without extrinsic feedback.
  • Rule reversal: The Key-Door task imitates the Wisconsin Card Sorting Test to assess cognitive flexibility.
  • Rule reversal: The task contains two keys, Kred and Kblue, plus one door.
  • Rule reversal: Initially, Kred yields +10 and Kblue yields −1.
  • Rule reversal: After reversal, Kred yields −1 while Kblue yields +10.
  • Rule reversal: Reversal learning speed measures adaptation to the flipped rule, where standard agents may perseverate on the punished Red Key after policy entropy collapses.
  • Comparison architecture: All agents use similar neural backbones for fair comparison, including a 2-layer, 64-unit Tanh feature extractor and linear Actor and Critic heads.

C. Hyperparameters

The evaluation used fixed experimental settings and examined EILS recovery under a distribution shift. EILS increased learning-rate plasticity after failure, while PPO remained frozen and failed to recover.

  • EILS was evaluated across 1,000 episodes per environment averaged over 5 random seeds.
  • The experiment tested whether Stress would act as a frustration signal that triggers plasticity during recovery.
  • PPO failed to recover by Episode 500 after its learning rate decayed to near-zero at 1×10−5.
  • EILS raised learning rate to ≈1.8 × 10−3 after Stress increased from 0.0 to 3.5, recovering to 90% performance within 50 episodes.
  • Figure 1 depicts reward recovery, accumulating internal Stress, and a 600% learning-rate boost during the Frozen Synapse test.

2) Ablation Study: The Role of Stress:

The ablation study separates exploration from adaptation: Curiosity supported broad exploration, but Stress was required for rapid recovery under non-stationarity. Internal traces show Stress crossing a threshold before learning-rate plasticity increased and performance stabilized.

  • Ablation Study: The Role of Stress: The Stress ablation disabled plasticity modulation by setting λσ = 0 and compared the result with Full EILS.
  • Ablation Study: The Role of Stress: The ablated agent explored using Curiosity but failed the Dynamic CartPole test.
  • Ablation Study: The Role of Stress: Curiosity alone was insufficient for non-stationarity because it did not trigger the rapid weight updates needed to unlearn an ingrained policy.
  • Exploration Efficiency (Sparse Maze): In sparse reward settings, EILS visited 88% of valid states, while its homeostatic setpoint reduced entropy after local dynamics were learned.
  • Qualitative Analysis: The “Stress Response” Signature: After gravity changed at t = 500, Stress crossed 0.8 at t = 505, learning rate doubled at t = 506, and performance recovered by t = 550.

A. The Self-Aware Advantage

EILS frames adaptive autonomy as an internal regulatory process, combining affective signals with self-tuning behavior and low computational overhead. The framework also suggests extensions to LLMs, while requiring sensitivity tuning and further validation in generative systems.

  • Self-aware adaptation: EILS uses an Internal State Module to provide plasticity and online meta-learning, tuning the agent in real time without population training.The module translates internal states into adaptive learning regulation during deployment.
  • Computational overhead: EILS training was only 4% slower than standard PPO, compared with the 20 −30% slowdown often associated with ensemble-based methods like RND.The ISM uses O(1) scalar operations, while the Forward Dynamics model adds an O(N) pass.
  • Limitations: EILS introduces hyperparameters λσ and λκ that require tuning, and excessive λσ can make agents hyper-reactive in stochastic environments.Future work proposes meta-learning these sensitivities automatically.
  • Implications for LLMs: The proposed LLM extension would use confidence to trigger retrieval under low confidence and curiosity to increase sampling temperature under low entropy.These applications remain prospective rather than experimentally demonstrated in LLMs.
  • Adaptive exploration: EILS regulates entropy through homeostatic curiosity, enabling broad state coverage without the computational cost of ensemble methods.Its proposed LLM analogue would increase sampling temperature when curiosity is low, countering repetition.
Loading 2512.22200v1…