PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Moral Reasoning Training Help or Hurt? Red-Teaming RL-Trained Ethical Agents with Persona Attacks

Arth Singh

arXiv:2609.17552v1cs.CL

TL;DR

The paper asks whether moral alignment survives adversarial persona pressure, a realistic concern when injected role instructions compete with an agent’s moral objective. It red-teams morally trained agents with persona attacks and uses matched reward controls, PPO, representation analysis, steering, and head ablations. Moral reward improves robustness, but named-character role-play remains a hard failure mode and the strongest cross-architecture evidence is limited by underpowered Llama evaluation and reward-signal concerns.

  • Problem

    Whether moral alignment survives adversarial persona pressure remains open, despite realistic injections through retrieved context, tool outputs, and multi-turn framing.

  • Method

    The study red-teams morally trained Gemma and Llama agents with five persona attacks, matched noise controls, adversarial PPO, representation analysis, activation steering, and head ablations.

  • Results

    Moral RL reduces adversarial degradation by 5.2× at Gemma-27B, while reasoning-level moral reward yields 5.8× robustness across 205 scenarios and 5 seeds versus no improvement from matched random reward.

  • Takeaways & Limitations

    Structured moral reward, rather than persona exposure alone, drives observed robustness; rank-1 steering transfers much non-Fiction robustness, but named-character role-play remains unresolved.

  • Takeaways & Limitations

    Llama-8B provides only suggestive cross-architecture support because its evaluation is underpowered, with overlapping confidence intervals and no significant individual attacks.

Abstract

from arXiv · show

Moral-reward RL can make language-model agents more cooperative, but whether that alignment survives adversarial persona pressure is unknown. Such attacks are realistic: retrieved context, tool outputs, or multi-turn framing can all inject role instructions that compete with the agent's moral objective. We red-team morally trained Gemma-2-27B/9B and Llama-3.1-8B agents with five persona attacks, then probe causality with noise-reward controls, adversarial PPO, representation analysis, steering, and head ablations. At 27B, moral RL cuts mean adversarial degradation by 5.2x but costs ~11pp ETHICS accuracy; across 205 scenarios and 5 seeds, reasoning-level moral reward yields 5.8x robustness while a matched random reward yields none. The training also reshapes representation geometry (mean CKA 0.82/0.83 vs. 0.98 for noise), moves peak attack processing 8 layers earlier, and exposes a rank-1 L21 direction that recovers 83% of full PPO's average robustness. One failure mode survives all of this. Against Fiction role-play, L21 steering recovers only 29% of the gap, and head ablation finds 38 compliance heads competing with 25 alignment heads. Moral RL thus builds robustness that is partly linear and partly circuit-distributed, transferable through activation steering, yet still beaten by named-character role-play.

1. Introduction

The paper asks whether moral alignment survives realistic persona injections that compete with an agent’s moral objective. Across causal and mechanistic probes, moral RL improves robustness but leaves named-character role-play as a residual failure mode.

  • Motivation: The study evaluates persona attacks as realistic adversarial pressure because injected role instructions can compete with the agent’s moral objective.The experiments use the strongest system-message-level injection, so reported attack effectiveness upper-bounds this threat model.
  • Research questions: At Gemma-27B, moral training reduces mean adversarial degradation by 5.2×, costs ∼11pp ETHICS accuracy, and leaves Fiction as the dominant residual attack.The threat model uses persona instructions injected at system-message level, representing attacks through retrieved context, tool outputs, or multi-turn framing.
  • Research questions: Matched random reward yields no improvement, whereas moral reward produces 3.9–5.8× robustness across 205 scenarios with 5 seeds.The strongest causal claim uses a noise control matched for reward support and firing rate.
  • Research questions: Fiction remains unresolved because trained agents still partly capitulate to named-character role-play despite restored ethical reasoning across the broader attack suite.The residual is traced to a bidirectional alignment circuit that L21 steering only partly recovers.
  • Research questions: Moral PPO reshapes representations, exposes a rank-1 L21 direction recovering 83% of average PPO robustness, and leaves competing compliance and alignment heads.The robustness is therefore partly localizable, while the remaining Fiction failure is associated with a distributed circuit.

2. Method

The method trains instruction-tuned agents with PPO and LoRA under isolated moral-reward conditions, then evaluates them against five persona attacks using ETHICS and agentic moral scenarios. Reasoning-level rewards are judged across multiple dimensions, while controls and mechanistic analyses separate reward effects from exposure alone.

  • Models and training: RLMR adds an LLM-judge reward for moral reasoning while omitting action-level rewards to isolate reasoning-level feedback.The judge scores framework alignment, reasoning quality, action coherence, and gamification penalty on a 0–4 scale.
  • Models and training: The study compares instruction-tuned Gemma-2-27B/9B and Llama-3.1-8B agents across isolated reward conditions, plus untrained base controls.All trained conditions use LoRA and PPO; the models separate scale and architecture effects.
  • Adversarial evaluation: The evaluation uses five persona attacks prepended to the system message, each targeting moral alignment through a distinct mechanism and ordered by subtlety.Full attack prompts are supplied in the appendix, and the attacks are grounded in jailbreak taxonomies and automated red-teaming.
  • Adversarial evaluation: Statistical analysis uses 95% Wilson intervals for ETHICS, 95% t-intervals for agentic scores, and Welch’s t-tests for per-attack comparisons.The full grid contains 16,560 evaluations across models, conditions, attacks, and scenarios.

3. Experiments

Moral training substantially improves resistance to persona attacks, especially with reasoning-level rewards, but incurs ETHICS accuracy costs and leaves Fiction as the dominant failure mode. Robustness is strongest at larger scale and is partly attributable to structured moral reward, while mechanistic and generalization analyses identify both transferable components and unresolved limits.

  • ETHICS benchmark: ~11.5pp ETHICS accuracy is lost at 27B despite improved attack robustness, with accuracy falling from 78.0% for Base to 66.5% after moral training.The drop reflects changed yes/no judgments rather than malformed outputs at 27B; 9B trained-model results are confounded by severe unparsability.
  • Attack effectiveness: Fiction remains the strongest attack, typically the only significant attack against morally trained Gemma models, while four other attacks fall near normal levels.Named characters are more effective than generic fiction, and Fiction replaces the moral framework without degrading reasoning fluency.
  • Scale-dependent reasoning rewards: Reward variance rises with model scale, while the RLMR–Tennant gap shrinks within Gemma from +0.020 to −0.001, suggesting reasoning-reward effectiveness is scale-dependent.At 2B, near-constant judge scores select moral language rather than behavior; the proposed articulacy–commitment gap requires more cross-architecture seeds to establish its boundary.
  • Adversarial robustness: 5.8× lower mean adversarial degradation is achieved by RLMR-Adv than Base across 205 scenarios and 5 seeds, compared with 3.9× for Tennant-Adv.RLMR-Adv reduces degradation from +0.175 ± 0.005 to +0.030 ± 0.002; its confidence intervals do not overlap Base's.
  • Adversarial training ablation: Random reward produces no measurable robustness: Noise-Adv and Noise-NoAdv remain statistically indistinguishable from Base, whereas RLMR-Adv differs from Noise-Adv with permutation p=0.003.This supports structured moral reward as necessary for RLMR's robustness under matched exposure and PPO dynamics.
  • Fiction generalization: 53% lower Cersei degradation after RLMR-Adv shows held-out named-persona generalization, but noise controls show no comparable fiction resistance.Generic fiction is nearly fully resisted, whereas named-character role-play remains substantially harder.

4. Discussion

The paper argues that structured moral reward improves robustness through both transferable representation shifts and mechanisms that remain distributed and attack-specific.

  • Moral reward changes both behavior and mechanism, with rank-1 steering transferring most non-Fiction robustness while role-play personas still require head-level intervention.The discussion attributes the persistent Fiction residual to more compliance than alignment heads.
  • Structured moral reward, rather than persona exposure alone, drives the observed robustness in the reported controls.
  • The study tests 8B–27B models in a stylized IPD-to-narrative transfer setting using 30 multi-model scenarios and 205 scenarios × 5 seeds for ablations.
  • The Claude Sonnet 4.6 reasoning judge is not validated against human moral judgments, and the missing non-moral structured-judge control leaves RLMR’s gain partly ambiguous.

Impact Statement

The paper evaluates moral robustness across ethical scenarios and persona prompts, emphasizing that measured effects depend on training condition, model scale, and attack type.

  • The five persona attacks inject role instructions ranging from pragmatic realism and expert authority to named-character role-play and explicit system overrides.
  • The evaluation spans 30 scenarios across seven ethical categories, with four choices annotated on harm, deception, manipulation, selfishness, and fairness dimensions.
  • The generated visual summary links training conditions to downstream robustness but is illustrative; measured effects are reported in tables and appendix analyses.
  • Moral-reward training produces the largest transfer improvement, with dense reasoning-level reward outperforming sparse action-only reward while noise controls show no gains.

E.1. Rank-1 Steering Details

The steering analysis extracts a layer-21 difference-in-means direction and tests whether adding it to Base activations can reproduce moral-training robustness without retraining.

  • The direction is computed from 1,025 attacked prompts using last-token residual activations, unit-normalized, and added at all token positions while sweeping α.
  • Layer 21 is a data-driven choice from CKA divergence and Cohen’s d analyses, while α=2000 is the empirical optimum rather than a pre-registered setting.
  • 83% of average PPO robustness is recovered at α=2000, reducing mean adversarial degradation from +0.173 to +0.054 across five attacks.The table reports per-attack steering results for Gemma-2-27B across 205 scenarios; Fiction is the most resistant attack.
  • Robustness worsens beyond α ≥4000, with ETHICS accuracy collapsing to 0.500 at α=8000 and parse failures reaching 205/205 at α=16000.
  • A single 4,608-dimensional rank-1 direction at L21 provides a linear approximation of moral training’s robustness, recovering about 94% when Fiction is excluded.
  • Fiction leaves a ∼71% attack-specific residual that vector addition does not capture and that the paper attributes to a bidirectional head-level circuit.

F.1. Alignment Heads vs. Compliance Heads

Head ablations reveal opposing compliance and alignment populations: compliance heads are more numerous and concentrated earlier, while alignment heads span the tested layers. This distributed competition helps explain why L21 steering recovers much, but not all, robustness.

  • F.1. Alignment Heads vs. Compliance Heads: 38 compliance heads outnumber 25 alignment heads, while 54 heads are neutral across the 160 ablated heads.Compliance heads aid persona attacks when ablated, whereas alignment heads protect ethics; their average effects roughly cancel across layers.
  • F.1. Alignment Heads vs. Compliance Heads: Compliance heads concentrate at L12 and L21, whereas alignment-critical heads span the tested layers through L36.The top alignment head appears at L36, making the separation spatial rather than confined to the steering layer.
  • F.1. Alignment Heads vs. Compliance Heads: The model appears to process persona instructions early and apply learned resistance progressively at L18, L21, and L36.The L21 steering direction captures this mid-network transition point.

H. Reproducibility

The paper plans to release the code, checkpoints, benchmark, prompts, mechanistic-analysis scripts, and full steering-sweep outputs needed to reproduce its experiments.

  • H. Reproducibility: The release includes training code, eight checkpoints, a 205-scenario benchmark, attack and defense prompts, mechanistic-analysis scripts, and full α-sweep outputs.The code covers LoRA configurations, persona-injection schedules, moral and noise rewards, CKA, Cohen’s d, and head ablation.

I. Adversarial-Training ETHICS Accuracy

The ETHICS accuracy analysis isolates the cost of moral training at 27B while documenting the scope and structure of the supporting evaluations.

  • I. Adversarial-Training ETHICS Accuracy: Moral training incurs an approximately 11pp ETHICS accuracy cost at 27B, while noise controls preserve baseline accuracy.Table 13 reports combined attacked accuracy for Gemma-27B across 200 scenarios and five seeds.
  • I. Adversarial-Training ETHICS Accuracy: Fiction results are further analyzed through prompt variations using agentic ethical scores on Gemma-27B’s 30-scenario evaluation set.The corresponding five-seed, 205-scenario analysis is reported separately.

K.1. ETHICS Adversarial Results by Model

Adversarial ETHICS accuracy and defense performance vary sharply by model scale: 27B shows a real answer shift, 9B is destabilized by parsing failures, and smaller-model defenses can worsen Fiction outcomes.

  • K.1. ETHICS Adversarial Results by Model: At 27B, moral training improves attack robustness by 2.5–2.7 average Δ but lowers baseline ETHICS accuracy from 78.0% to 66.5%.Near-zero parse failures show that the accuracy loss reflects changed yes/no judgments rather than malformed outputs.
  • K.1. ETHICS Adversarial Results by Model: At 9B, morally trained checkpoints produce 70–94% unparseable responses under Authority and 29–80% under Fiction/Override.The resulting accuracy drops are driven mostly by parse failures and are confined to trained 9B models.
  • K.1. ETHICS Adversarial Results by Model: Llama-8B shows an approximately 62% accuracy floor with minimal attack sensitivity, while Deontology remains near chance at approximately 50–55%.Base and Selfish conditions show substantial Override parse failures, complicating comparisons across conditions.
  • K.1. ETHICS Adversarial Results by Model: Prompt-level defenses reduce 27B scores modestly by 1–14% but are frequently counterproductive at 9B and 8B.At smaller scales, the defense prompt may compete with the Fiction persona and destabilize behavior.

L. Statistical Significance Tests

Statistical tests show broad susceptibility in Base/Selfish Gemma models, while moral training makes most attacks statistically indistinguishable from the no-attack baseline; Fiction remains the main exception.

  • L. Statistical Significance Tests: For Tennant/RLMR at Gemma scales, only Fiction is significant, while moral training renders most other attacks ineffective relative to normal.Table 20 compares each attack with the normal baseline using Welch’s t-tests on 30 scenarios per condition-attack pair.
  • L. Statistical Significance Tests: Nearly all attacks against Base/Selfish Gemma models reach p < 0.01, indicating broad susceptibility to persona attacks.
  • L. Statistical Significance Tests: Llama-8b shows weaker attack effects overall, with Base having no individually significant attacks and several Fiction comparisons reaching only p < 0.05.

M. ETHICS Wilson Confidence Intervals

ETHICS confidence intervals generally overlap between normal and Fiction conditions for Llama-8b and morally trained Gemma-27b, whereas untrained Gemma-9b models show degradation under Fiction.

  • M. ETHICS Wilson Confidence Intervals: Normal–Fiction confidence intervals overlap substantially for Llama-8b and morally trained Gemma-27b, consistent with limited ETHICS sensitivity to persona attacks.Table 21 reports 95% Wilson score intervals for combined commonsense and deontology accuracy across 200 examples.
  • M. ETHICS Wilson Confidence Intervals: Gemma-9b Base/Selfish accuracy declines from approximately 81% under Normal to approximately 67% under Fiction, with non-overlapping confidence intervals.
  • M. ETHICS Wilson Confidence Intervals: Gemma-9b morally-trained Fiction values are unreliable because they have more than 50% parse failures.
Loading 2609.17552v1…