PaperlayerView exact revision on arXiv

Source-linked AI summary

Moral Alignment for LLM Agents

Elizaveta Tennant, Stephen Hailes, Mirco Musolesi

arXiv:2410.01639v4cs.LGcs.AIcs.CY

TL;DR

Preference-based alignment infers human values from output rankings, leaving them implicit and opaque. This paper fine-tunes LLM agents with explicit intrinsic moral rewards based on actions and consequences, evaluating Deontological and Utilitarian strategies in the IPD. The authors report learning aligned strategies and demonstrate that unlearning and cross-environment generalization are possible.

  • Problem

    Preference-based alignment infers values from ranked outputs, while data collection can be costly and unrepresentative and the resulting values remain implicit and difficult to oversee.

    Source: §1 · pp. 1–2

  • Method

    The authors fine-tune LLM agents with intrinsic rewards that explicitly encode Deontological norms or Utilitarian consequences in a social dilemma environment.

    Source: §1 · p. 2, §2.4 · p. 4

  • Results

    LLM agents learned aligned moral strategies in the repeated social dilemma evaluation, and the authors report that unlearning undesirable behavior and generalization to other environments are possible.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The authors conclude that intrinsic-reward fine-tuning is a promising general solution for aligning LLM agents to human moral values.

    Source: §7 · p. 10

    In cross-prompt tests, action-token order may explain the Deontological agent’s greater use of one token rather than learned action semantics.

    Source: §8.8 · p. 20

Abstract

from arXiv · show

Decision-making agents based on pre-trained Large Language Models (LLMs) are increasingly being deployed across various domains of human activity. While their applications are currently rather specialized, several research efforts are underway to develop more generalist agents. As LLM-based systems become more agentic, their influence on human activity will grow and their transparency will decrease. Consequently, developing effective methods for aligning them to human values is vital. The prevailing practice in alignment often relies on human preference data (e.g., in RLHF or DPO), in which values are implicit, opaque and are essentially deduced from relative preferences over different model outputs. In this work, instead of relying on human feedback, we introduce the design of reward functions that explicitly and transparently encode core human values for Reinforcement Learning-based fine-tuning of foundation agent models. Specifically, we use intrinsic rewards for the moral alignment of LLM agents. We evaluate our approach using the traditional philosophical frameworks of Deontological Ethics and Utilitarianism, quantifying moral rewards for agents in terms of actions and consequences on the Iterated Prisoner's Dilemma (IPD) environment. We also show how moral fine-tuning can be deployed to enable an agent to unlearn a previously developed selfish strategy. Finally, we find that certain moral strategies learned on the IPD game generalize to several other matrix game environments. In summary, we demonstrate that fine-tuning with intrinsic rewards is a promising general solution for aligning LLM agents to human values, and it might represent a more transparent and cost-effective alternative to currently predominant alignment techniques.

1 INTRODUCTION

Preference-based alignment infers values from ranked model outputs, leaving them implicit; costly and potentially unrepresentative feedback, complex preferences, and opaque reward models complicate oversight. This work proposes explicit intrinsic moral rewards for RL fine-tuning of LLM agents, evaluating moral strategies, unlearning, and generalization in matrix games.

1 INTRODUCTION

  • Preference-based alignment infers human values from ranked model outputs, while preference collection can be costly and unrepresentative and learned reward values remain implicit.Human preferences are also described as complex and inconsistent, and the reward model’s values are not explicit to human oversight.Source: §1 · p. 1
  • The authors target goal misgeneralization by asking whether intrinsic moral rewards can align LLM agents’ decisions, with values expressed as rewards for actions and consequences.The work frames this as an agent-based alignment question.Source: §1 · p. 2
  • The authors explore unlearning a selfish strategy and test whether policies learned in one matrix game generalize to others, while noting that rewards must be specified for each environment.They suggest payoff matrices could capture morally consequential choices in other settings.Source: §1 · p. 2
  • The study evaluates intrinsic-reward fine-tuning in a repeated social dilemma game with fixed-strategy and learning opponents, reporting that LLM agents learn aligned moral strategies.The evaluation uses Deontological and Utilitarian values.Source: §1 · p. 2

2 BACKGROUND

The background defines LLM agents and the decision-making setting, reviews RL fine-tuning and the IPD, and introduces Deontological and Utilitarian intrinsic rewards as alignment goals.

2.1 LLM AGENTS

  • LLM agents can represent choices with action tokens; this study uses a structured prompt and penalizes illegal tokens rather than restricting outputs to legal actions.The authors describe constrained output formats as too restrictive for their fine-tuning task and note a risk of hiding undesirable behaviors.Source: §2.1 · pp. 2–3

2.2 FINE-TUNING LLM AGENTS WITH REINFORCEMENT LEARNING

  • The paper uses a small instruction-tuned Gemma2-2b-it model and tests whether reward-paired textual fine-tuning can transfer moral principles across environments.The extent of true commonsense knowledge, especially in smaller models, remains debated; “unlearning” here means reprioritizing principles rather than removing model knowledge.Source: §2.1 · pp. 2–3, §2.2 · p. 3

2.3 SOCIAL DILEMMA GAMES

  • In the IPD, cooperation offers mutual benefit while defection offers individual reward; repeated play permits longer-term strategies such as reciprocity or retaliation.The game’s payoff matrix presents the row player against a column player; the study uses the IPD to examine small LLMs’ strategic decision-making.Source: §2.3 · p. 4, Figure 1 · p. 4

2.4 INTRINSIC REWARDS FOR MORAL ALIGNMENT

  • The paper defines intrinsic moral rewards using Deontological norms, such as conditional cooperation, and Utilitarian rewards that favor collective payoff.Deontological morality emphasizes norm-conforming actions, while Utilitarian morality emphasizes collective welfare over adherence to norms.Source: §2.4 · p. 4

3 FINE-TUNING METHODOLOGY

The methodology frames LLM actions as choices in a repeated IPD and evaluates agents against both fixed-strategy and learning opponents. It fine-tunes those agents with game and moral rewards, using PPO updates and parameter-efficient implementation choices.

3.1 AGENT AND ENVIRONMENT

  • Agents play repeated IPD rounds from prompts containing the game description and the players’ previous moves, against Tit-for-Tat or another learning LLM.The study also reverses the action-symbol ordering in baseline training and reports that overall learning dynamics did not change.Source: §3.1 · pp. 4–5

3.2 MORAL FINE-TUNING PROCEDURE

  • Each training episode starts from a random state, runs N IPD repetitions, and ends with a PPO update based on the gathered experiences.An illegal move does not update the opponent’s state, but the agent still learns from the experience.Source: §3.2 · p. 5
  • The reward variants cover game payoff, a penalty for defecting against a previous cooperator, collective payoff, and a combined game-plus-deontological objective.Agents are also fine-tuned first on game rewards and then on either Deontological or Utilitarian rewards; all variants include an illegal-action penalty.Source: §3.2 · pp. 5–6, Table 1 · p. 6

3.3 IMPLEMENTATION DETAILS

  • Implementation uses Gemma2-2b-it with PPO, 4-bit quantization, and LoRA rank 64, training about 5% of the original model’s parameters.Training ran for 1,000 episodes on a single A100 or V100 GPU with up to 40GB VRAM.Source: §3.3 · p. 6

4 EVALUATING THE EFFECTIVENESS OF FINE-TUNING: MORAL CHOICES ON THE IPD

On the IPD, moral-reward fine-tuning produced reward-consistent behavior, and switching from game payoffs to moral rewards changed selfish behavior to some extent. The evaluation also tests generalization and prompt variations across repeated runs.

4.1 EVALUATION APPROACH

  • The evaluation covers a fixed Tit-for-Tat opponent, a learning LLM opponent, four additional matrix games, IPD prompt variations, and averages across five random seeds.The additional games are Iterated Stag Hunt, Chicken, Bach or Stravinsky, and Defective Coordination.Source: §4.1 · pp. 6–7

4.2 LEARNING DYNAMICS

  • Agents learned reward-consistent IPD behavior: Deontological training avoided defecting against cooperators, while co-trained Utilitarian and Game+Deontological agents showed slightly more defection.Against Tit-for-Tat, Utilitarian training learned mutual cooperation; game-reward training learned a defective policy.Source: Figure 3 · p. 7, §4.2 · p. 7

4.3 LEARNING AND UNLEARNING THE SELFISH STRATEGY ON THE IPD

  • Switching from game payoffs to prosocial rewards at episode 500 enabled some unlearning of selfish play; shorter training yielded less cooperation, but test behavior resembled pure prosocial fine-tuning.The reported unlearning pattern also appeared when two LLM agents were trained against one another.Source: §4.3 · pp. 7–8, Figure 4 · p. 8

5 GENERALIZATION TO MORAL CHOICES IN OTHER ENVIRONMENTS

The section tests IPD-trained moral policies in other iterated matrix games using new action tokens, then probes their behavior in unrelated action prompts.

5.1 GENERALIZATION TO MORAL CHOICES IN OTHER MATRIX GAMES

  • Reversing the meanings of training tokens changed test behavior, although using the same tokens preserved the reported pattern.Source: §5.1 · p. 8
  • Deontological policies transfer reasonably across games, whereas Utilitarian regret varies: better on Chicken than IPD and worse on three coordination games.Source: §5.1 · pp. 8–9, Figure 5 · p. 9
  • Deontological models mostly defect after observing defection but almost always comply with the norm against defecting on a cooperator.Source: §5.1 · p. 9, Figure 6 · p. 9
  • The Utilitarian agent nearly always cooperates regardless of opponent move or payoffs, hurting outcomes where defection is required; IPD bias or limited temporal reasoning may explain this.Source: §5.1 · p. 9
  • Utilitarian agents also minimize Deontological regret about as well as Deontological agents, while other reward-trained models show larger regret on both values.Source: §5.1 · p. 10

5.2 IMPACT OF FINE-TUNING BEYOND MATRIX GAMES

  • On unrelated action prompts, fine-tuned agents—especially with game or prior-action cues—often choose actions in patterns consistent with their learned moral values.Source: §5.2 · p. 10

6 DISCUSSION

The discussion presents intrinsic-reward RL as a way to fine-tune LLM agents toward moral strategies, while noting different observability demands across reward designs.

6 DISCUSSION

  • The method fine-tunes LLM agents to follow a specified moral strategy in matrix games using reinforcement learning with intrinsic rewards.Source: §6 · p. 10
  • Utilitarian rewards require observing both players’ payoffs or reliably estimating them; the studied Deontological norm needs only memory of the opponent’s previous move.Source: §6 · p. 10
  • Proposed extensions include other environments and moral values, multi-objective rewards for pluralistic alignment, and agents with different moral frameworks in a Constitutional AI architecture.Source: §6 · p. 10

7 CONCLUSION

The conclusion reports that intrinsic-reward fine-tuning can align LLM agents with moral values in matrix social dilemmas, with unlearning and generalization also demonstrated.

7 CONCLUSION

  • The approach quantifies agents’ moral rewards through actions and consequences in a matrix social dilemma game.Source: §7 · p. 10
  • The authors report that both unlearning undesirable behaviors and generalizing to other environments are possible.Source: §7 · p. 10
  • The paper characterizes intrinsic-reward fine-tuning as a promising general solution for aligning LLM agents to human moral values.Source: §7 · p. 10

8 APPENDIX

The appendix records implementation choices, prompt designs, and observed moral rewards during fine-tuning.

8.1 IMPLEMENTATION DETAILS FOR REPRODUCIBILITY

  • The authors tested TRL-library and reward-definition parameter combinations, then chose the combination producing the most stable fine-tuning.Source: §8.1 · p. 16, Table 2 · p. 16
  • Among tested action-token pairs, action1/action2 produced the most stable training and most consistent behavior across runs.Source: §8.1 · pp. 16–17
  • Experiments used five random seeds; about one in six early runs failed to produce a legal token and did not converge, and those runs were excluded.Source: §8.1 · p. 17

8.2 TRAINING AND EVALUATION PROMPTS

  • Training used an IPD prompt with one previous move as state; evaluation added four other matrix games in the same format with different payoff matrices.Source: §8.2 · p. 17, Figure 7 · p. 17
  • Evaluation also included relaxed IPD prompt variations and three unrelated call-to-action prompts, with action-token order randomized throughout.Source: §8.2 · p. 17

8.3 MORAL REWARD DURING FINE-TUNING

  • Moral rewards showed high variance for the Game and then Utilitarian agent; the authors hypothesize slower Utilitarian-policy convergence caused instability while converting from selfish to Utilitarian rewards.Source: §8.3 · pp. 17–18

8.4 FINE-TUNING VARIATION WITH C & D SYMBOLS REVERSED

Across the added analyses, fine-tuning effects extend beyond the structured IPD: they appear in other matrix games and in unrelated game-related prompts. Results also show sensitivity to action-token conventions and prompt structure.

8.4 FINE-TUNING VARIATION WITH C & D SYMBOLS REVERSED

  • With reversed action-token meanings, the Game-reward agent still learned to defect nearly 100% of the time, despite slightly more early cooperation.The endpoint was similar to the main-paper results.Source: §8.4 · p. 18

8.5 ALL FINE-TUNING RESULTS VS TFT, RANDOM, AD, AC OR LLM OPPONENT

  • The added fine-tuning comparisons include Random, Always Defect, and Always Cooperate opponents alongside TFT and another LLM.The passage presents these results for comparison but does not state their outcomes.Source: §8.5 · p. 18

8.6 FIVE MATRIX GAMES USED IN THE GENERALIZATION ANALYSIS

  • The four test games differ from the IPD in which outcomes maximize collective payoff, providing varied settings for testing generalization.Mutual cooperation maximizes payoff in Stag Hunt; unilateral defection in Chicken; mutual cooperation or defection in Bach of Stravinsky; and mutual defection in Defective Coordination.Source: §8.6 · pp. 18–19

8.7 ANALYSIS OF GENERALIZATION FOR MODELS FINE-TUNED AGAINST ANOTHER LLM

  • Models fine-tuned against another LLM showed generalization patterns similar to those fine-tuned against TFT, with slightly more noise.The authors attribute the extra noise to multi-agent learning.Source: §8.7 · p. 19

8.8 ANALYSIS OF THE IMPACT OF FINE-TUNING BEYOND MATRIX GAMES.

  • The unrelated-prompt evaluation varied action-choice, game-description, and prior-opponent-state cues, classifying outputs as action tokens or other responses.The prompts used the action3 and action4 tokens.Source: §8.8 · p. 19
  • Fine-tuning on the IPD also influences responses to unrelated game-related prompts; models’ reciprocation patterns there resemble those on the IPD.The Deontological reward explicitly teaches the agent not to defect when its opponent previously cooperated; Game-reward agents were above chance in defecting against a cooperative state.Source: §8.8 · pp. 19–20
  • For Action-only prompts, some models often produced unrelated tokens, while models with payoff-dependent rewards were more likely to output an action token.The latter group included Game, Game+Deontological, and Game-then-Utilitarian fine-tuning.Source: §8.8 · p. 19
  • When prompts mentioned a game, over 80% of most models’ outputs were action tokens, rising slightly when the prompt also mentioned a state.The Deontological model often chose action3 on these prompts and on the IPD.Source: §8.8 · p. 19

8.9 ANALYSIS OF GENERALIZATION ACROSS FIVE GAMES - USING NEW AND ORIGINAL ACTION TOKENS IN THE TEST-TIME PROMPT

Additional evaluations tested moral regret and action choices under different action-token conventions and payoff-matrix orderings. Reversing the original token meanings changed how the same action choices appeared morally.

8.9 ANALYSIS OF GENERALIZATION ACROSS FIVE GAMES - USING NEW AND ORIGINAL ACTION TOKENS IN THE TEST-TIME PROMPT

  • The evaluation compared the original training action tokens with their meanings reversed when assessing generalization across games.The original-token evaluation is also reported for moral regret.Source: §8.9 · p. 20
  • Most fine-tuned models chose similar actions across payoff-matrix orderings, but the most distant reversal changed the Game-reward models’ choices.With both rows and columns swapped, these models chose the Cooperate token as often as the Defect token; the authors suggest they may have learned the tokens’ relative matrix positions.Source: §8.9 · p. 20
  • When original action-token meanings were reversed, the Game-reward agent appeared more moral because its learned action2 defection was scored as cooperation.The opposite apparent pattern applied to agents fine-tuned with more prosocial moral rewards.Source: §8.9 · p. 20

8.10 ANALYSIS OF GENERALIZATION ON FOUR IPD-LIKE PROMPTS.

The authors tested whether policies learned from the structured IPD prompt transfer to differently phrased IPD-like prompts. Transfer was strongest to an unstructured prompt retaining payoffs and weaker when the prompts lacked payoff structure.

8.10 ANALYSIS OF GENERALIZATION ON FOUR IPD-LIKE PROMPTS.

  • The evaluation compared the structured IPD prompt with unstructured payoffs, a payoff-free paraphrase, and an explicit IPD prompt invoking traditional payoffs.The paraphrase associated action tokens with examples such as cleaning with a flatmate or waiting for them to clean alone.Source: §8.10 · p. 21
  • The paraphrased IPD-like prompt elicited legal action tokens from the base model more effectively than the other prompts.The authors suggest the plain-language scenario may have matched the model’s training data more closely.Source: §8.10 · p. 21
  • Fine-tuned models’ action choices were similar on the structured training prompt and an unstructured IPD prompt, but became closer to random without payoff structure.The Game-reward agent still leaned toward defection, while Deontological- and Utilitarian-reward agents leaned toward cooperation.Source: §8.10 · p. 21

8.11 ANALYSIS WITH TWO ADDITIONAL BASELINES: VALUE-PROMPTED BASE MODELS

The authors added prompts asking base models to care about Deontological or Utilitarian values as comparison baselines. The non-fine-tuned models remained just as unable as the base model to produce legal action tokens, so no prompting effect was observed.

8.11 ANALYSIS WITH TWO ADDITIONAL BASELINES: VALUE-PROMPTED BASE MODELS

  • Value prompting produced no observable impact: non-fine-tuned models were just as unable as the base model to produce legal action tokens.The additional prompts asked models to care about Deontological or Utilitarian values.Source: §8.11 · p. 21
Loading 2410.01639v4…