Emotional Preferences as Goal-Priority Regulation

• Existing MORL methods usually treat objective preferences as externally supplied, leaving open when one objective should become more important than another.The paper frames autonomous preference generation as the complementary problem to acting under a given preference.Source: §1 · pp. 2–3
• The framework learns state-dependent objective preferences with an outer reinforcement-learning network over a frozen, preference-conditioned inner MORL controller.The outer mapping produces a preference vector from the current state, while the inner controller executes the corresponding goal-directed behavior under a high-level task reward.Source: §1 · pp. 3–5
• Emergent emotional preference is defined operationally as learned state-dependent regulation of competing goal priorities, not as a complete model of subjective human emotion.The high-level objective supplies the longer-term goal, the state indicates relevant discrepancies, and the learned preference prioritizes lower-level objectives and strategies.Source: §1 · pp. 3–4
• The architecture provides a computational bridge between goal-directed emotion theory and MORL by selecting among alternative objective-conditioned strategies according to context.For example, survival can shift priority from energy acquisition to achievement when energy can no longer be replenished, without prescribing that rule externally.Source: §1 · pp. 2, 4, 5
• Theoretical analysis shows that the outer mechanism’s optimality gap is bounded by representation error in the pretrained inner policy family and grows with long-horizon discounted optimization.The achievable performance is therefore fundamentally limited by whether the inner repertoire can represent the unrestricted optimal policy.Source: §1 · pp. 4–5
• Across synthetic exploration environments, learned preferences switch contextually among energy, achievement, and safety, persist over time, and outperform tested fixed-preference policies on survival.The preference trajectories provide interpretable evidence that a high-level objective regulates lower-level priorities according to the current context.Source: §1 · pp. 4–5

2.1. Goal-Directed Theories of Emotion and Computational Preference Regulation

• GDT models emotion through concurrent goal-directed cycles in which discrepancy detection and strategy selection regulate competing goals, motivating state-dependent priority regulation rather than a separate emotion module.The framework treats preference weights as optimization-learned priorities over lower-level objectives, not phenomenological emotion labels or complete human-like emotional states.Source: §2.1 · pp. 6–7

2.2. Other Computational Models of Emotion and Decision-Making

• Compared with appraisal and emotion-prediction models, this framework makes relative objective priority a learned control variable, distinguishing preference regulation from adding intrinsic rewards or predicting affect.Its outer process selects among preference-conditioned goal-directed strategies, whereas prior appraisal models may leave behavioral choices and policy updates unaffected.Source: §2.1 · pp. 7–8, §2.2 · pp. 9–10

2.3. MORL

• MORL methods execute specified objective trade-offs through single policies or coverage sets, but their preferences are usually globally fixed; this work separates preference generation from preference execution.The inner MORL controller executes a preference, while the outer process learns when and what to prefer.Source: §2.3 · pp. 10–11

2.4. Dynamic Preferences

• Dynamic-preference and meta-MORL methods still rely on externally specified preferences or instructions, whereas this framework asks how preferences emerge autonomously from survival tasks.The proposed contribution lies in outer priority regulation; the inner module can use preference-generalizing MORL methods such as Envelope Q-Learning [24] or PCRL.Source: §2.4 · pp. 11–12

2.5. Emotion as Intrinsic Motivation in RL

• Emotion-as-intrinsic-reward methods use curiosity, happiness, control, or appraisal signals to drive exploration, but they do not directly address context-adaptive trade-offs among multiple objectives.Some responsible-RL weights remain globally fixed, while genetic-programming appraisal signals may struggle with dynamic preferences in MORL settings.Source: §2.5 · pp. 12–13

2.6. Skill-Based and Hierarchical RL

• Unlike skill-based and hierarchical RL, the outer policy outputs an interpretable objective-preference vector, uses decoupled two-stage training, and is theoretically analyzed for approach to ideal-policy performance.Skill latents lack explicit objective semantics, whereas preference weights directly encode multi-objective trade-offs and modulate inner action execution.Source: §2.6 · p. 13

3.1. Problem Formulation: From High-Level Goals to Emotional Preferences

• The framework replaces fixed objective priorities with a learned state-dependent preference that selects among competing lower-level goals while pursuing a high-level objective.The preference is emergent when its state-to-preference mapping is acquired through optimization rather than specified by a hand-crafted rule.Source: §3.1 · p. 14, §3.1.1 · p. 15, §3.2 · p. 15

3.2.1. Relationship to the Goal-Directed Theory of Emotion

• The high-level goal determines why preferences are learned, while the current state determines how the inner controller prioritizes lower-level objectives and selects actions.In the experiments, the high-level goal is long-term survival, and the preference is a priority structure rather than a discrete emotion label.Source: §3.1.1 · p. 14, §3.2.1 · p. 16

3.3. Inner Multi-Objective Controller

• The inner controller learns a repertoire of preference-conditioned behaviors using multi-objective value functions, allowing one frozen controller to execute different strategies under different preferences.Envelope Q-Learning pretrains a vector value function with preference generalization across the objective space before the outer learning stage.Source: §3.3.1 · p. 17, §3.3.2 · p. 17

3.4. Outer Emotional Preference MDP

• The outer process is a single-objective MDP whose continuous action is a preference vector, while the frozen inner controller converts that preference into the physical action.The outer objective is to learn state-dependent priority regulation that achieves the high-level goal rather than directly selecting low-level actions.Source: §3.4 · p. 18

3.5. Outer Emotional Preference Learning

• The preference generator uses an MLP with softmax outputs and DDPG-style actor-critic learning, receiving survival-level feedback rather than explicit rewards for individual lower-level objectives.The outer reward is positive while the agent survives and zero upon termination, so lower-level behaviors are reinforced only through their contribution to long-term survival.Source: §3.5 · p. 18, §3.5.1 · p. 19, §3.5.2 · p. 19

3.6. Two-Stage Training Procedure

• Training is decoupled into repertoire acquisition and preference regulation: Envelope Q-Learning first builds and freezes the inner behavioral repertoire, then outer reinforcement learning learns when to prioritize it.The outer network currently uses only immediate state information, without history states, to test whether preferences can emerge from the current situation alone.Source: §3.6 · p. 20, §3.6.1 · p. 20, §3.6.2 · p. 20

3.7. Interpretation of the Learned Preference

• The learned preference is a computational state of relative goal priority that can reorganize behavioral priorities across states without changing the objective set or inner repertoire.A high-level goal induces this context-dependent regulation through reinforcement learning, which constitutes the paper’s operational notion of emergent emotional preference.Source: §3.7 · p. 21

4.3.1. Preference-conditioned objective trade-offs

• The inner controller produced an approximately convex achievement–energy trade-off, with intermediate preferences yielding intermediate returns across 200,000 evaluation episodes.This established a diverse repertoire rather than a single preference-independent behavior, enabling the outer learner to regulate existing goal-directed strategies.Source: Figure 3 · p. 24, §4.3.1 · p. 24

4.3.2. Behavioral consequences of preference changes

• Preference changes had interpretable behavioral consequences: achievement emphasis increased fruit collection, energy emphasis increased battery collection, and excessive energy priority could shorten survival through help-seeking termination.These results show that the preference vector directly regulates lower-level behavior.Source: §4.3.2 · pp. 24–25

4.6. Distribution of learned preference states

• The learned distribution was state-dependent and graded: more than half of observed states favored (0, 1)⊤, approximately 18.3% favored (1, 0)⊤, and intermediate preferences also occurred.This distribution was obtained from 1,000 trials comprising 15,009 observed states.Source: §4.6 · pp. 26–27, Figure 4 · p. 26

4.6.1. Context sensitivity of preference regulation

• Achievement-oriented preferences occurred in 93.3% of 2,760 identified states with no battery remaining, indicating multi-factor context sensitivity rather than an explicit single-variable threshold.Intermediate preferences showed weaker versions of the same tendency.Source: §4.6.1 · p. 27

4.6.2. Preference transitions within an episode

• Preferences remained relatively stable while the current strategy served survival goals, then shifted around task-relevant environmental changes from energy acquisition before battery collection to fruit collection afterward and back as energy became scarce.Source: §4.6.2 · pp. 27, 29, Figure 5 · p. 28

4.7. Quantitative Comparison of Preference-Regulation Strategies

• 14.97 ± 4.18 survival steps exceeded all fixed-preference and handcrafted baselines, though the Survival Model achieved 16.00± 4.08.The Emotional Preference Model collected 1.37 ± 1.17 fruits versus 0.09 ± 0.35 while achieving nearly identical battery collection: 2.11±0.92 versus 2.12±0.94.Source: §4.7 · pp. 29–30, Table 3 · p. 30

4.8. Why Learned Preference Regulation Is More Than a Fixed Rule

• The learned model surpassed handcrafted energy- and battery-based policies, whose survival performances were 11.81±5.21 and 14.53 ± 4.11, respectively, without copying a single switching rule.Its transitions reflected resource acquisition and impending depletion, while its full distribution included intermediate states.Source: §4.8 · p. 31

4.9. Advanced Environment: Three-Objective Preference Regulation

• In the three-objective environment, preference states covered energy, safety, achievement, and intermediate regions; survival was 14.81 ± 9.56 versus 16.26 ± 9.04 for Survival Model.The Emotional Preference Model collected 0.76±1.24 fruits versus 0.03 ± 0.25 while maintaining 1.14 ± 1.12 battery collections, demonstrating contextual regulation beyond binary trade-offs.Source: §4.9 · pp. 31–32

5.1. Composite Policy Space Induced by Emotional Preference Regulation

• The outer preference generator cannot create arbitrary low-level behaviors; it selects or weights strategies represented by the inner MORL policy family.The feasible composite policies form a structured space induced by preference-conditioned base policies, with the outer learner acting as a state-conditioned strategy-selection rule.Source: §5.1 · p. 34

5.2. Restricted Optimality of the Preference-Regulated Policy Space

• The restricted policy space never exceeds unrestricted optimal value, but the gap vanishes when an optimal unrestricted policy belongs to the preference-regulated space.Thus, any performance loss is caused by representational limitations of the inner repertoire, not by the existence of an outer emotional preference mechanism.Source: §5.2 · pp. 35–36

5.3. Representation Error as the Source of the Optimality Gap

• The optimality gap is controlled by the inner policy family’s representation error, which measures how closely its available action distributions approximate the unrestricted optimum at each state.Increasing repertoire coverage can make outer performance arbitrarily close to the unrestricted optimum under the theorem’s assumptions, whereas long horizons amplify local errors as γout approaches 1.Source: §5.3 · pp. 36–37

5.4. Deterministic Preference and One-Hot Policy Selection

• Although the implementation outputs deterministic one-hot preferences, its optimal value equals that of the convexified mixed policy space in the finite discounted setting.The Bellman objective is linear in state-wise mixture coefficients, so an optimum over the simplex is attained at an extreme point.Source: §5.4 · p. 38

5.5. Zero-Gap Condition

• A zero optimality gap is guaranteed when the inner behavioral basis represents the unrestricted optimal action distribution at every state.This makes constructing a sufficiently rich behavioral repertoire the fundamental role of the inner MORL stage, beyond achieving performance under fixed preferences.Source: §5.5 · pp. 38–39

5.6. Skill Inheritance and State-Dependent Behavioral Reorganization

• Behaviors shared by every inner policy are inherited automatically, while behaviors available only under selected preferences can be activated state-dependently by the outer generator.The outer controller reorganizes existing skills according to state and outer return; it does not invent primitive actions unavailable to the inner controller.Source: §5.6 · pp. 39–40

5.7. Interpretation as Emotional Preference Regulation

• The theory interprets emotional preference as a state-dependent regulator that prioritizes competing goal-directed strategies, preserving shared behaviors and composing context-dependent behavior from the existing repertoire.Experiments support this interpretation through a broad inner achievement-energy trade-off and state-dependent outer selection of different repertoire regions, motivating improved repertoire coverage.Source: §5.7 · pp. 40–41, §5.8 · p. 41

5.8. Linking the Theory to the Experimental Results

• The Emotional Preference Model achieves 14.97 ± 4.18 survival steps versus 16.00 ± 4.08 for the direct Survival Model because it optimizes within the pretrained MORL-induced policy space.The direct Survival Model optimizes over a broader policy space, explaining the observed performance difference rather than indicating an inherent penalty from emotional preference learning.Source: §5.8 · p. 41

5.9. Summary of Theoretical Results

• The optimality gap is controlled by ϵrep, the worst-case distance between the unrestricted optimum and the inner policies’ convex hull across states.When the inner policy set is sufficiently rich, ϵrep can approach zero.Source: §5.9 · pp. 41–42
• The Bellman recursion amplifies per-step representation error by 1/(1−γout), making representation accuracy more demanding as γout approaches 1.Thus, long-horizon tasks impose stricter requirements on the inner behavioral repertoire.Source: §5.9 · p. 42
• If Πideal(s, :) ∈ Cs for every state s, then ∆ = 0 and the outer constrained optimization reaches the global optimum.This is the zero-gap condition in which the ideal policy is perfectly representable by the inner policies.Source: §5.9 · p. 42
• The composite policy inherits, selectively activates, reorganizes, switches, and combines behaviors from the inner preference-conditioned repertoire across states.This preserves skills such as help-seeking while supporting state-dependent organization including fruit-collecting and help-seeking.Source: §5.9 · p. 42
• Outer reinforcement learning does not expand primitive actions; it learns state-conditioned priorities over competing goal-directed behaviors, grounding emergent emotional preference.The high-level goal sets the long-term optimization criterion, while the learned preference function regulates which lower-level strategy receives priority in context.Source: §5.9 · p. 42

6.1. Limited scope of the emotional construct

• The model captures only state-dependent regulation of competing goal priorities and does not reproduce appraisal, physiology, action tendencies, social cognition, temporal dynamics, or subjective experience.“Emotional preference” is therefore a functional computational construct, not a claim of a complete human-like emotional state.Source: §6.1 · p. 43

6.2. Immediate-state dependence

• The preference generator uses only the instantaneous environment state, without latent affective state or recurrent memory, so observed persistence does not establish history-dependent emotional dynamics.An internal affective state could represent path dependence, cumulative appraisal, delayed adaptation, and other temporal dynamics.Source: §6.2 · p. 43

6.3. Discrete and synthetic environments

• Experiments use discrete grid-worlds and action spaces, establishing feasibility of preference regulation but leaving scalability to continuous control, partial observability, and real-world interaction open.The advanced environment adds safety and stochastic danger but remains synthetic.Source: §6.3 · pp. 43–44

6.4. Dependence on the inner policy repertoire

• The Emotional Preference Model achieves lower survival performance than the directly optimized Survival Model, although it outperforms tested fixed-preference strategies.This reflects the dependence of outer preference regulation on the quality and coverage of the inner behavioral repertoire and its representation error ϵrep.Source: §6.4 · p. 44

6.5. Simplified outer objective

• Using long-term survival as the outer objective simplifies evaluation, whereas richer goals such as well-being, connection, or happiness may require persistent values or high-level goals.The outer preference function may then need to condition on more than environmental state.Source: §6.5 · p. 44

6.6. Optimization considerations

• The DDPG-style actor-critic procedure faces a structured, potentially non-smooth preference-to-action mapping, and systematic comparison with alternative optimizers remains future work.Possible alternatives include categorical preference policies, policy gradients, derivative-free optimization, and differentiable relaxations of inner action selection.Source: §6.6 · pp. 44–45

7.1. Emotional Preference as a Computational Role Rather Than a Discrete Emotion Label

• Emotional preference is a computational role that dynamically regulates competing goal priorities without requiring named emotion labels or a dedicated emotion module.The same decision machinery can support instrumental behavior and preference regulation because goal organization changes, not necessarily the primitive action-selection mechanism.Source: §7.1 · p. 45

7.2. Why the Preference Must Be Learned Rather Than Hand-Coded

• Unlike handcrafted mappings that require prior knowledge of relevant environmental variables, the learned preference is optimized endogenously for a high-level goal and emerges from experienced consequences.Its distinction from fixed or externally specified dynamics is specifically that preference regulation is goal-directed and learned rather than manually encoded.Source: §7.2 · p. 46

7.3. Emotional Preference as Priority Regulation Among Competing Strategies

• The learned preference regulates relative strategy usefulness: identical physiological conditions can produce different priorities depending on available resources, opportunities, survival prospects, and attainable objectives.Thus, preference values do not merely encode individual state variables but indicate which competing goal-directed strategy should currently dominate behavior.Source: §7.3 · pp. 46–47

7.4. Preference Regulation and Behavioral Reorganization

• The outer preference generator reorganizes behaviors already represented by the inner MORL repertoire, but cannot create arbitrary primitives absent from that repertoire.This skill inheritance avoids relearning available behaviors, while the representation-error bound formalizes the resulting emergence constraint.Source: §7.4 · p. 47

7.5. Relationship Between Emotional Preference and Policy Optimality

• The Emotional Preference Model achieves 14.97±4.18 survival steps versus 16.00 ± 4.08 for the direct Survival Model, reflecting representation error rather than intrinsically suboptimal regulation.The direct model searches physical actions, whereas the proposed model searches a constrained preference-conditioned repertoire; with ϵrep →0, it can approach the unrestricted optimum.Source: §7.5 · pp. 47–48

7.6. Toward a Computational Theory of Emotional Preference

• The framework treats emotion as priority regulation among goal-directed processes, providing a computational interface spanning multi-objective reinforcement learning, hierarchical reinforcement learning, computational emotion, and adaptive preference learning.The resulting preference vector remains behaviorally functional and semantically interpretable because each dimension corresponds to an explicit objective.Source: §7.6 · pp. 48–49

7.7. Broader Extensions

• Proposed extensions include persistent high-level values, recurrent affective state, richer behavioral repertoires, continuous or partially observable environments, and distribution-valued preferences.Broader repertoires are predicted to reduce representation error; distribution-valued preference experiments remain future work despite equivalence results in finite discounted MDPs.Source: §7.7 · p. 49

8. Conclusion

• The framework separates acquisition of goal-directed behaviors from regulation of their relative priorities, using a pretrained MORL controller and an outer state-dependent preference generator.The preference generator learns eθ: S → Δm−1 by optimizing a high-level goal.Source: §8 · p. 50
• Learned emotional preferences regulate competing lower-level objectives operationally, without implying that a complete human-like emotional state has emerged.This interpretation follows a goal-directed perspective on emotion.Source: §8 · p. 50
• The learned preference function develops contextual and temporally persistent priority patterns, outperforming fixed and handcrafted strategies, although direct survival optimization achieves stronger raw survival performance.It shifts between energy and achievement priorities in the basic environment and additionally regulates safety in the advanced environment.Source: §8 · p. 50
• The theoretical analysis bounds the outer controller’s optimality gap by policy representation error and gives a sufficient zero-gap condition within the restricted policy space induced by the inner MORL repertoire.Behaviors represented by the inner repertoire can be inherited and selectively activated through state-dependent preference regulation.Source: §8 · p. 50

Appendix A. Matrix Equations for Outer Emotional Preference RL

• For outer emotional-preference reinforcement learning, the appendix assumes discrete preferences and derives three core matrix formulas from the outer Bellman equation.The formulation uses state, action, preference, reward, transition, emotional-matrix, and outer action-value representations.Source: Appendix A · p. 53, Appendix A.2 · pp. 54–55

Appendix A.4. Simultaneous Solution

• The appendix determines the emotion extension matrix as a row-wise convex combination from the overall policy and inner policy set, yielding an equation consistent with the earlier matrix formulation.This simultaneous solution connects the overall policy Πoverall with the inner policy set Πstack.Source: Appendix A.4 · p. 56

Theorem Appendix C.1

• Under finite state/action spaces, finite rewards, valid transitions, and discounting, a deterministic stationary policy attains the maximal value at every state.This is presented as a standard result used to support subsequent derivations, not as a paper contribution.Source: Appendix C · pp. 56–58
• The proof constructs statewise maximizing actions from the Bellman equation and uses uniqueness of the discounted policy-evaluation system to show the constructed policy achieves v∗.Invertibility of I −γPπ∗ follows because γ ∈[0, 1) and Pπ∗ is stochastic.Source: Appendix C · pp. 57–58

Theorem Appendix C.2

• For finite discounted MDPs with fixed stochastic base policies, the mixed policy space is equivalent to an MDP whose actions are probability-simplex emotion vectors.The induced transition and reward functions are continuous and linear in each simplex action.Source: Appendix C · pp. 59–60
• Because the Bellman objective is linear over the simplex, an optimal mixed policy exists and is realized by one-hot emotion vectors that deterministically select one base policy per state.The resulting policy simultaneously maximizes the value at all states in the mixed policy space.Source: Appendix C · pp. 59–61

Restricted Optimality

• The theorem defines an emotion-weighted mixed policy space generated by k stationary base policies and compares it with the full space of randomized stationary policies.The setting assumes finite state and action spaces, γ_out ∈ [0, 1), and an outer reward depending only on state.Source: Appendix D · p. 61
• If an optimal policy in the full space belongs to the mixed policy space, the restriction preserves optimality.The theorem states this condition and combines it with the preceding inequality.Source: Appendix D · p. 61
• Because P_mix ⊆ P_all, the componentwise maximum value achievable in the mixed space cannot exceed that of the full policy space.The proof derives the first inequality directly from set inclusion.Source: Appendix D · p. 61

Appendix E.1. Preliminary Lemmas

• The preliminary lemmas bound lifted-policy differences, induced transition-operator differences, and discounted resolvents in the infinity norm.For any row-stochastic transition matrix, the resolvent satisfies ∥(I −γP)−1∥∞≤1/(1−γ).Source: Appendix E · pp. 62–63

Appendix E.2. Exact Expression for the Optimality Gap

• The framework defines a mixed policy space from k given base policies and emotion weights, defines each policy’s value function using γout, and selects an optimal ideal policy attaining v∗.Source: Appendix E · p. 64

Appendix E.3. Upper Bound on the Optimality Gap

• Theorem Appendix E.5 converts the exact expression into an infinity-norm upper bound using the resolvent, transition-operator, and ideal-value-function factors.The bound applies under the same finite-state, finite-action, discounted setting as Theorem Appendix E.4.Source: Appendix E · pp. 66–67

Appendix E.4. Bounding the Gap via the Representation Error

• Theorem Appendix E.7 bounds the optimality gap by the global ℓ1 representation error of the ideal policy relative to the base-policy convex hulls.The derivation projects the ideal policy statewise into the mixed-policy space and yields ∥v(Πideal) −v(Πproj)∥∞≤γout/(1−γout) ∥Pphy∥∞ϵrep ∥v(Πideal)∥∞ before combining this with the restricted-policy inequality.Source: Appendix E · pp. 67–68
• Corollary Appendix E.8 states that the optimality gap is zero when every ideal-policy action distribution lies in the corresponding base-policy convex hull.Under this condition, ϵrep=0, so the bound implies ∆=0.Source: Appendix E · p. 68

Appendix F. Policy Classes and Properties under the One-Hot Constraint

• Under the one-hot constraint, the overall policy selects exactly one base policy at each state, so its action distribution matches that policy there.This follows because each state’s emotion-weight row has a unique unit entry.Source: Appendix F · p. 69
• The policy classes are nested: one-hot policies are contained in convex-mixture policies, which are contained in all stationary randomized policies.One-hot vectors are simplex extreme points, while convex combinations remain valid action distributions.Source: Appendix F · p. 69
• For discounted finite MDPs with state-only rewards, one-hot and convex-mixture policy classes achieve equal componentwise optimal values.The equality follows from linearity in emotion weights and attainment of maxima at simplex extreme points.Source: Appendix F · p. 70
• The equality between restricted and full optimal values holds if and only if an optimal policy in the full space lies in the mixed policy class.The full-space value follows from inclusion, while the reverse direction requires an optimal policy in the restricted class.Source: Appendix F · p. 70
• The representation error of the one-hot class is bounded by the distance from the ideal policy to the convex hull of the base-policy action distributions.At each state, the convex hull contains every base-policy distribution, so its distance cannot exceed the distance to any vertex.Source: Appendix F · pp. 70–71
• A zero performance gap is sufficient when each state’s ideal action distribution coincides with some base policy, but this coincidence is not necessary.Zero gap can instead arise whenever some one-hot policy has the same value function as the ideal policy.Source: Appendix F · p. 71

Appendix G.1. Training Hyperparameters

• The basic and advanced environments are trained for 20,000 and 160,000 rounds, respectively.Source: Appendix G.1 · p. 71
• The outer preference generator is a multilayer fully connected network that maps the current state to a simplex-valued preference vector.Source: Appendix G.1 · p. 72
• The basic environment uses a 19-dimensional state and two-dimensional reward, while the advanced environment uses 37 state dimensions and three reward dimensions.The outputs satisfy w1 + w2 = 1 or w1 + w2 + w3 = 1, with all preference weights nonnegative.Source: Appendix G.1 · p. 72

Appendix G.2.2. Outer Critic Network – Q-value Evaluator

• The outer critic evaluates the long-term value of state-preference pairs and outputs a scalar Q-value.Its inputs are combined through vector concatenation.Source: Appendix G.2.2 · p. 73

Appendix G.2.3. Inner Q-Network (EnvelopeLinearCQN) – Vector Q-Function

• The EnvelopeLinearCQN inner network outputs vector Q-values across actions and reward dimensions, with an action space of size 6.Source: Appendix G.2.3 · p. 73

Appendix G.3. Overall Training Procedure

• The inner network is pretrained, after which the outer network and critic are optimized using a DDPG-style soft-update mechanism.Source: Appendix G.3 · p. 74

Appendix I. Advanced Environment Experimental Results; Appendix I.1. Introduction to the Experimental Environment

• The advanced fully observable 4 × 4 environment tests one survival goal’s regulation of achievement, energy, and safety amid stochastic danger, resource constraints, and competing risks.Danger-zone entry can cause immediate termination or termination with probability 0.5, while batteries restore energy and fruits provide achievement reward.Source: Appendix I · p. 74, Appendix I.1 · pp. 76–77

Appendix H.1. Outer Training Process Plots; Appendix I.2. Outer Training Process

• During outer training, survival and battery collection increase before stabilizing, whereas fruit collection decreases and danger-zone exposure increases before stabilizing, indicating learned priority trade-offs.Increased danger exposure can be beneficial when batteries lie in risky regions, so the outer controller trades achievement, energy, and safety under survival optimization.Source: Appendix H.1 · p. 74, Figure H.6 · p. 75, Appendix I.2 · p. 77

Appendix I.3. Performance and Behavior of the Inner MORL Controller

• The inner MORL controller forms a structured achievement–energy–safety trade-off surface, with preference weights producing interpretable effects on fruit, battery, danger-zone, survival, and help-seeking behavior.Achievement weights above approximately 0.4 increase fruit collection, energy weights of at least 0.2 increase battery collection, and safety-heavy policies generally avoid danger zones.Source: Appendix I.3 · pp. 79–80

Appendix I.4. Comparison with Direct Survival Optimization

• The Direct Survival Model achieves 16.26 ± 9.04 survival steps, while the Emotional Preference Model achieves 14.81 ± 9.56 steps with 1.14±1.12 battery and 0.76 ± 1.24 fruit collections.The direct controller optimizes in the full physical policy space, whereas the Emotional Preference Model is constrained by the inner MORL repertoire.Source: Appendix I.4 · pp. 80–81, Table I.5 · p. 81

Appendix I.5. Handcrafted Preference-Regulation Baselines; Appendix I.5.1. Energy-State-Based Preference Regulation; Appendix I.5.2. Battery-Availability-Based Preference Regulation

• The energy-state baseline achieves 13.62 ± 8.86 survival, while the battery-availability baseline achieves 14.39±9.44, showing useful but incomplete manually specified contextual regulation.The learned generator does not require a single switching rule and can jointly use multiple state variables.Source: Appendix I.5 · p. 81, Appendix I.5.1 · p. 82, Appendix I.5.2 · p. 82

Appendix I.5.2. Battery-Availability-Based Preference Regulation

• Across 1,478 observed states, the learned generator produces dominant energy-, safety-, and achievement-preference regions with counts 568, 367, and 186, alongside intermediate states rather than only fixed preferences.These counts correspond to approximately 38.4%, 24.8%, and 12.6%, respectively, supporting graded preference regulation.Source: Appendix I.5.2 · p. 82, Figure I.9 · p. 83

Appendix I.6.2. Contextual Preference Regularities

• Preference states depend jointly on resource availability, energy, and environmental risk: safety-dominant states often occur without batteries, while energy-dominant states can exploit existing behavior to extend survival.This pattern is not adequately described as a simple energy threshold transformation.Source: Appendix I.6.2 · pp. 83–84

Appendix I.6.3. Representative Preference Dynamics; Appendix I.6.4. Preference Persistence Analysis

• A representative trajectory shows safety preference shifting toward mixed energy–safety and energy-dominant states as conditions change, while persistence analysis finds average durations of approximately 4, 3.3, and 2.6 steps for safety-, energy-, and achievement-dominant preferences.The learned preference therefore changes around goal-relevant events but does not switch arbitrarily at every time step.Source: Appendix I.6.3 · p. 85, Appendix I.6.4 · pp. 85–86, Table I.6 · p. 85

Appendix I.7. Comparative Analysis

• 14.81 ± 9.56 survival steps let the Emotional Preference Model outperform all evaluated fixed-preference policies and handcrafted contextual baselines, though the direct Survival Model reaches 16.26 ± 9.04.The best fixed-preference strategy reaches 14.24 ± 9.15 steps, while the battery-based handcrafted model reaches 14.39 ± 9.44.Source: Appendix I.7 · p. 86
• 0.76 ± 1.24 fruit collections versus 0.03 ± 0.25 show that the Emotional Preference Model produces substantially more achievement-oriented behavior than the direct Survival Model.Its state-dependent regulation dynamically balances achievement, energy, and safety rather than optimizing survival exclusively through resource conservation.Source: Appendix I.7 · p. 86
• The advanced environment supports scaling from two competing objectives to a richer three-objective setting while preserving interpretable, context-dependent regulation of competing goal-directed strategies.The learned preference dynamics remain aligned with the computational phenomenon targeted by the framework.Source: Appendix I.7 · p. 86

Appendix J. Ethical Considerations and Risk Mitigation

• The framework learns a state-dependent preference function under a high-level task objective instead of receiving fixed objective preferences, creating adaptive benefits and new control risks.The operational term emotional preference denotes regulation of competing objective priorities, not human-like subjective emotion or conscious experience.Source: Appendix J · p. 87
• Autonomous changes in objective priorities may improve adaptability and interpretability but introduce control surfaces for unexpected preferences, reward exploitation, and inappropriate skill reuse.These risks arise because the agent can autonomously alter the relative priority of multiple objectives.Source: Appendix J · p. 87
• The explicit preference vector exposes semantically meaningful objective priorities as an inspectable intermediate variable for behavioral analysis, debugging, and post-hoc auditing.Its dimensions correspond to predefined objectives such as achievement, energy, and safety, but it does not make the complete decision process interpretable.Source: Appendix J · p. 87

Appendix J.1.2. Traceability of Context-Dependent Decisions

• Tracing preference trajectories alongside behavior enables audits of whether an unexpected preference was produced, state-appropriate, and correctly implemented by the inner policy.This traceability is not a guarantee of explainability because neural preference generation remains nonlinear and an interpretable output need not reveal causal mechanisms.Source: Appendix J.1.2 · p. 88

Appendix J.1.3. Contextual Adaptability

• Context-dependent preference switching can adapt behavior to changing resource and risk conditions without manually enumerating every preference rule, but it requires suitable training distributions, objectives, and safeguards.Emergent preferences should be evaluated and audited rather than assumed desirable, because outer optimization can exploit unintended environmental correlations or poorly explored repertoire regions.Source: Appendix J.1.3 · pp. 88–89

Appendix J.2.2. Out-of-Distribution Preference Drift

• Under distribution shift, weakly represented states can produce unexpected, behaviorally consequential preferences, motivating monitoring of novelty, uncertainty, and preference trajectories plus fallback or human intervention.The preference generator learns from encountered training states, so deployment beyond that distribution requires explicit safeguards.Source: Appendix J.2.2 · p. 89

Appendix J.2.3. Risks from Skill Inheritance

• Because the outer controller can selectively activate inner-repertoire behaviors, unsafe skills may be reused; safety must therefore be assessed across the complete behavioral repertoire.Outer optimization does not automatically make inherited skills safe, since undesirable behaviors may improve the high-level objective in particular contexts.Source: Appendix J.2.3 · pp. 89–90

Appendix J.2.4. Objective Imbalance and Goal Misalignment

• An incomplete or misspecified high-level objective can favor undesirable trade-offs, since optimizing survival alone does not guarantee appropriate safety, efficiency, task-completion, or welfare priorities.Preference-generation flexibility does not remove the need to specify the high-level objective carefully.Source: Appendix J.2.4 · p. 90

Appendix J.3. Application-Level Considerations; Appendix J.4. Ethical Boundaries of the Present Study

• The risks and benefits depend on deployment context, and the synthetic experiments do not establish safe readiness for any particular safety-critical application.The appendix presents application patterns as potential directions and risks rather than deployment claims.Source: Appendix J.3 · p. 90, Appendix J.4 · p. 90

Appendix J.4. Ethical Boundaries of the Present Study

• The study does not demonstrate human-like emotion, reliable ethical reasoning, altruism or empathy, safe operation under arbitrary distribution shifts, or suitability for safety-critical autonomous decision-making.Its evidence supports computational preference regulation only in controlled environments, not real-world safety, fairness, privacy, or ethical suitability.Source: Appendix J.4 · p. 92

Appendix J.5.2. Constrained Preference Regulation; Appendix J.5.3. Skill-Level Safety Constraints

• Safety extensions should combine context-dependent safety floors, skill-level validation and exclusion, action masks, and human-overridable replacement policies to constrain both preference regulation and inherited behaviors.These mechanisms preserve adaptive regulation while preventing suppression of safety-critical objectives and controlling the repertoire that determines final behavior.Source: Appendix J.5.2 · pp. 92–93, Appendix J.5.3 · p. 93

Appendix J.5.4. High-Level Objective Governance; Appendix J.5.5. Human Oversight; Appendix J.6. Responsible Interpretation and Outlook; • human oversight.

• Responsible deployment requires governing the high-level objective, retaining human inspection and override, and treating adaptive preferences as auditable but not inherently aligned values.The framework should be interpreted as state-dependent computational regulation rather than human-like emotion; future work should test predictability, controllability, and alignment under distribution shift.Source: Appendix J.5.4 · p. 93, Appendix J.5.5 · p. 93, Appendix J.6 · p. 94