Source-linked AI summary
Emotion in Reinforcement Learning Agents and Robots: A Survey
Thomas M. Moerland, Joost Broekens, Catholijn M. Jonker
TL;DR
Emotion in RL agents lacks a unifying framework spanning machine learning, affective modelling, and human-robot interaction, while the review’s scope excludes most human-user emotion. The paper systematically surveys emotion elicitation, types, functions, and evaluation, finding distinct but overlapping computational categories and identifying social communication and reproducibility as important directions.
Problem
Emotion-in-RL research spans disconnected communities and lacks a full framework organizing models and implementations, while human emotional teaching signals remain a limited adjacent literature.
Method
The paper conducts a systematic survey organized by emotion elicitation, emotion type, emotion function, and evaluation criteria within a motivated-RL framework.
Results
The survey identifies homeostasis, appraisal, value/reward, and hard-wired connections as emotion-elicitation categories, with emotion functions spanning reward, state, value, action selection, and communication.
Takeaways & Limitations
Emotion models may inform intrinsic motivation, exploration, metaparameter tuning, and communication of internal learning processes in RL agents and robots.
Takeaways & Limitations
The survey focuses on agent emotions and does not further pursue human emotion as a teaching signal; included implementations also sometimes use distance-to-goal information unavailable to baselines.
Abstract
from arXiv · showhide
This article provides the first survey of computational models of emotion in reinforcement learning (RL) agents. The survey focuses on agent/robot emotions, and mostly ignores human user emotions. Emotions are recognized as functional in decision-making by influencing motivation and action selection. Therefore, computational emotion models are usually grounded in the agent's decision making architecture, of which RL is an important subclass. Studying emotions in RL-based agents is useful for three research fields. For machine learning (ML) researchers, emotion models may improve learning efficiency. For the interactive ML and human-robot interaction (HRI) community, emotions can communicate state and enhance user investment. Lastly, it allows affective modelling (AM) researchers to investigate their emotion theories in a successful AI agent class. This survey provides background on emotion theory and RL. It systematically addresses 1) from what underlying dimensions (e.g., homeostasis, appraisal) emotions can be derived and how these can be modelled in RL-agents, 2) what types of emotions have been derived from these dimensions, and 3) how these emotions may either influence the learning efficiency of the agent or be useful as social signals. We also systematically compare evaluation criteria, and draw connections to important RL sub-domains like (intrinsic) motivation and model-based RL. In short, this survey provides both a practical overview for engineers wanting to implement emotions in their RL agents, and identifies challenges and directions for future emotion-RL research.
1 Introduction
The survey frames emotion in RL agents as computationally functional for decision-making, learning, and social interaction. It establishes a framework connecting emotion models across reinforcement learning, affective modelling, and human-robot interaction.
- Motivation: Emotion models in RL agents are grounded in decision-making architectures and may persist after learning converges.The survey treats emotions as hooked to MDP characteristics such as value and transition functions, rather than as quantities the agent must learn.
- Motivation: Studying emotions in learning agents may benefit learning efficiency, communicate internal state, and support user empathy.These motivations connect the topic to intrinsically motivated RL, interactive machine learning, and HRI.
- Emotion and RL: The survey addresses emotion as a response linked to motivation, action tendencies, and feedback signals shaping behaviour.This framing connects emotional responses to both motivated action and behavioural regulation.
- Survey aims: The review aims to provide a framework connecting diverse emotion models and implementations in learning agents.It also serves as a practical guideline for engineers implementing emotional functionality in RL-based agents and robots.
- Survey aims: The survey organizes its review through background, methodology, emotion elicitation, emotion types, emotion functionality, and evaluation criteria.The main results systematically categorize approaches across these dimensions.
2 Background
The background positions emotion in RL within psychological, neuroscientific, and computational perspectives. It distinguishes major emotion theories, clarifies the RL setting, and limits the review primarily to agent rather than human emotions.
- Emotion theories: The survey contrasts categorical, dimensional, and componential theories as different ways of representing emotions.Categorical theories use discrete basic emotions, dimensional theories use affective spaces such as valence and arousal, and componential theories use appraisal patterns.
- Emotion theories: These theories emphasize different aspects of emotion, including elicitation, action tendencies, and feedback within reward-based loops.The survey therefore considers both emotion elicitation and emotion function.
- Neuroscience: Neuroscientific accounts link emotional responses to reward processing, bodily signals, and pathways that can bypass neocortical reasoning.The discussed theories include fear conditioning, somatic markers, and orbitofrontal involvement in reinforcement and affect.
- Related work: RL implementations of emotion are situated within a broader affective-modelling field that includes robotic, virtual-agent, symbolic, and other machine-learning approaches.The survey distinguishes learning-focused RL implementations from symbolic and cognitive architectures.
- Scope: The review focuses on agent emotions that influence learning loops, while largely excluding human emotion as a teaching signal.Human emotional feedback has been reported to speed learning in a grid-world task, but this literature is described as limited.
- Computational RL: Reinforcement learning is presented as a machine-learning framework in which agents adapt through value-function or policy-based methods.The survey focuses mainly on value-function methods while defining RL as sampling-based learning for sequential decision problems.
3 Survey structure and methodology
The survey uses a systematic literature search and organizes findings through a taxonomy of emotion elicitation, emotion type, and emotion function. Its motivated-RL schematic connects these categories to the learning loop.
- Methodology: The literature search combined keyword queries, abstract screening, full-article assessment when needed, and citation searching around core papers.The stated goal was to include research in which both reinforcement learning and emotion play a role.
- Taxonomy: The proposed taxonomy classifies papers by emotion elicitation, emotion type, and emotion function.Table 1 records the associated discussion sections, while Figure 1 visualizes elicitation and function categories.
- Taxonomy: The survey parallels extrinsic motivation with homeostasis and intrinsic motivation with appraisal as an organizing assumption.The authors explicitly describe these pairings as overlaps observed in computational implementations rather than synonyms.
- Motivated RL framework: The motivated-RL schematic places an internal Critic between sensations and the agent, integrating motivations into state and reward computation.The Agent then learns a value function and selects actions; ordinary externally rewarded RL is a special case.
4 Emotion elicitation
The survey identifies four major emotion-elicitation categories in RL agents: homeostatic, appraisal-based, value/reward-based, and hard-wired. These approaches derive emotions from internal variables, agent-environment interaction, value or reward signals, or fixed stimuli, while exposing implementation challenges for complex cognitive and social concepts.
- Four major emotion-elicitation categories are identified: extrinsic/homeostatic, intrinsic/appraisal, value-function or reward-based, and hard-wired.
- 4.1 Homeostasis and extrinsic motivation: Homeostatic models represent internal variables, drives, and motivations relative to set points, with drives commonly computed as weighted distances from those set points.The distance may use the L1 norm or another metric, and variable weights can change over time.
- 4.1 Homeostasis and extrinsic motivation: Homeostatic systems can derive categorical emotions or overall well-being, and their reward modifications naturally support goal and task switching.Well-being may itself represent emotion, or changes in well-being may represent happiness, unhappiness, or hedonic value.
- 4.2 Appraisal and intrinsic motivation: Appraisal implementations frequently model novelty, motivational relevance, and intrinsic pleasantness from learned transition functions, recency, or forward planning over model space.The survey connects these dimensions to intrinsic motivation and notes that a single concept can have substantially different implementations.
- Current RL algorithms have difficulty learning complex social phenomena, while abstract cognitive concepts remain difficult to translate into broadly accepted mathematical expressions.Some appraisal dimensions require cognitive reasoning and are therefore harder to implement.
- 4.3 Value and reward functions: Value- and reward-based emotion models comprise value-based, temporal-difference-based, average-reward-based, and raw-reward-based approaches.Some implementations derive valence from short- versus long-term average reward, temporal reward changes, or the reward signal itself.
- 4.4 Hard-wired: Hard-wired elicitation can derive emotions directly from stimuli such as collisions, but the survey finds that this category alone offers limited insight into emotion generation.The surveyed examples may nevertheless combine ideas from multiple elicitation methods.
5 Emotion type
The survey finds that categorical emotions dominate emotion-RL research, while dimensional models are less common but offer continuous affective representations. Similar emotions can arise from homeostatic, appraisal, and value-based elicitation methods, though implementations are often underspecified.
- Categorical emotions: Categorical emotions are more common than dimensional emotions, with joy or happiness the most frequently implemented category.Happy, sad, fear, and anger appear across homeostatic, appraisal, and value-based elicitation categories.
- Categorical emotions: Happy, sad, fear, and anger are associated with all three elicitation categories, whereas hope is mainly linked to value-function systems.Hope implementations assess anticipation through value functions, decision-cycle dynamics, or forward simulation of expected temporal differences.
- Elicitation methods: Different elicitation methods can produce similar emotions, and individual papers may combine multiple methods for one emotion.For example, fear can combine extrinsic pain with intrinsic or appraisal-based novelty.
- Elicitation methods: Many studies do not fully specify how elicitation methods connect to emotion types, making their implementations difficult to replicate.Some papers mention fuzzy logic or leave the implemented connections unspecified without describing the underlying principles.
- Dimensional emotions: Valence is the most implemented dimensional emotion, while arousal is the second most implemented and has been linked to homeostatic, appraisal-like, and hard-wired variables.Some studies use related terms such as well-being, relaxation, or restlessness instead of valence or arousal.
- Dimensional emotions: Dimensional models are under-represented, but their continuous structure remains attractive for engineering applications such as defining target regions in affective space.Valence shows more implementation consistency than arousal and other dimensions.
6 Emotion function
Emotion can enter the RL loop through reward, state, adaptive-critic parameters, or action selection, and can also remain an epiphenomenal social signal. The surveyed implementations include both hand-designed and learned emotional connections, with some approaches relying on additional information unavailable to baselines.
- Functional roles: Emotion-based RL implementations modify reward, state, adaptive-critic parameters, or action selection, while some only elicit emotions as social signals.The survey treats these as distinct functional roles within or outside the learning loop.
- Reward modification: Reward-modification approaches add an emotion-dependent internal reward to the external reward, implementing reward shaping.The internal reward may target positive emotions or depend on homeostatic and appraisal variables.
- Reward modification: Homeostatic reward systems make reward depend on internal variables, so obtaining a resource may yield no reward when the relevant variable is already satiated.This links reward to changes in well-being or internal drives rather than resource acquisition alone.
- Reward modification: Appraisal-based reward modifications use history- and goal-related variables such as novelty, recency, motivational relevance, and intrinsic pleasantness.Novelty bonuses correspond in RL terminology to optimism in the face of uncertainty.
- Limitations: The survey notes that some emotion-based appraisal systems use distance-to-goal information unavailable to their baseline agents.This additional information complicates interpretation of their empirical comparisons.
- State representation: Some emotion-state models include emotion in the state representation, enabling emotion-specific value functions and policies.Maggie, for example, incorporates elicited fear into the state-space and learns fear-specific action values.
- Adaptive critic and action selection: Emotion can steer RL metaparameters such as the temporal-difference error, discount factor, action-selection temperature, and learning rate.Different surveyed systems associate emotions including joy, anger, fear, relief, and surprise with these parameters.
7 Evaluation
Evaluation spans embodiment, scenarios, learning efficiency, emotion dynamics, and human-robot interaction. Although emotional functionality is often associated with improved learning outcomes or interaction measures, comparisons are constrained by small state-spaces and largely bespoke test scenarios.
- Embodiment: The survey evaluates implementations across five embodiment categories: standard agents, multiple agents, screen agents, simulated robots, and real robots.Most implementations use simulated agents, while real robots appear in navigation, resource, and human-interaction tasks.
- Embodiment: Most implementations use relatively small state-spaces and narrow or preprocessed sensory inputs, leaving scalability to complex high-dimensional problems unresolved.Small state-spaces facilitate interpretation but may not represent realistic sensing demands.
- Test scenarios: Test scenarios include navigation with resources or obstacles, smaller groups of competitive or cooperative multi-agent settings, and human-agent or robot interaction.Navigation tasks dominate the surveyed scenarios.
- Test scenarios: Most papers construct their own scenarios rather than reusing benchmarks, making implementations difficult to compare directly.The survey reports no borrowed scenarios from other emotion-learning implementations or the broader RL literature.
- Empirical outcomes: Evaluation outcomes fall into learning efficiency, emotion dynamics, and human-robot interaction.These categories distinguish performance, psychological correspondence, and social-interaction measures.
- Empirical outcomes: Emotional agents have achieved higher average rewards, learned faster, avoided collisions or obstacles, and switched between goals more effectively across reported tasks.These findings summarize varied author-reported outcomes rather than a common benchmark comparison.
- Empirical outcomes: Emotion-dynamics studies compare emergent signals with psychological patterns such as habituation, extinction, blocking, approach, and avoidance.Some studies also qualitatively assess whether dynamics fit social interaction.
- Empirical outcomes: Human-robot interaction studies report improved communication, greater agent persuasiveness, or stronger perceived connection in participant evaluations.These outcomes are typically assessed through post-experiment questionnaires.
8 Discussion
The survey organizes emotion in RL agents by elicitation, type, and function, while identifying evaluation, integration, reproducibility, and social-interaction challenges. It also offers implementation guidance and highlights connections between affective modelling and RL.
- Emotion elicitation: Emotion elicitation methods include homeostasis, appraisal, value or reward functions, and hard-wired implementations, with the first three addressing different aspects of emotion.Homeostasis concerns inner resource status, appraisal concerns inner model status, and value or reward concerns the learning process; these categories can overlap.
- Emotion types: Categorical emotion implementations are more common than dimensional models, which are often limited to one or two dimensions such as valence or valence-arousal.The survey hypothesizes that dimensional and categorical emotions could coexist at different levels within one system.
- Emotion function: Emotion can function at nearly every point in the RL loop, including reward, state, value functions, and action selection, while communicative emotional signals remain under-evaluated.The survey connects emotional communication to transparency of the learner’s internal process and identifies it as a future research direction.
- Implementation advice: The survey advises engineers to choose emotion mechanisms according to the desired functionality, such as visualizing agent state, enhancing empathy, or increasing user investment.For communicative or believable emotions, it recommends treating emotion as an epiphenomenon and consulting the corresponding elicitation and type categories.
- Testing and integration: The field needs integrated approaches, more reproducible system descriptions, and stronger evaluation practices across shared scenarios.The authors report that emotional scenarios had not been reproduced by other researchers and that implementation details were often insufficient for reproduction.
- Future directions: Social interaction remains an important open direction because current implementations seldom investigate full interaction involving human feedback, human emotions, and agent communication.The survey also identifies integrating psychological concepts into mathematical expressions and solving high-dimensional social settings as continuing challenges.
9 Conclusion
The survey organizes emotion models in RL agents by elicitation, emotion type, and function, and reports benefits for learning, adaptation, interaction, and emotion dynamics. It concludes by identifying implementation and evaluation opportunities across these areas.
- Future work: The survey identifies future work in implementing and evaluating emotion models in RL agents.Its evaluation overview distinguishes learning efficiency, emotion dynamics, and HRI criteria as separate targets.
- Learning: Emotion models improved learning outcomes across tasks, including quicker goal achievement, faster policy learning, fewer trials, and higher average return.The reported evaluations include both task-performance gains and improved learning efficiency.
- Learning: Emotional control of exploration improved time to goal and time to find the global optimum, while emotional control of simulation depth improved average return.These results connect emotion to both exploration and model-based planning choices.
- Emotion dynamics: Emotion dynamics reproduced or plausibly modeled habituation, extinction, blocking, anticipation, hedonic experience, and decay in several scenarios.The survey compares these dynamics with psychological theory or evaluates whether they occur appropriately in task scenarios.
- Behavior and interaction: Emotional agents showed behavioral effects including cooperation, adaptation to environmental change, obstacle avoidance, approach, and avoidance.Different emotions could produce different paths or alter behavior in interactive and embodied tasks.