Source-linked AI summary
Training and Evaluating Ethical Reinforcement Learning Agents on Per-Episode Distributions
Prabhjyot Singh, Majid Ghasemi, Mark Crowley
TL;DR
Ethical RL agents can appear compliant on average while concentrating violations in a few episodes, even though harms do not necessarily compensate across episodes. This paper compares four training objectives in Craftax using uncensored per-episode violation evaluation and finds that they match on the mean frontier but separate sharply in violation distributions, with ESR maintaining its budget at no mean-return cost.
Problem
Mean-based evaluation can miss ethical RL agents that concentrate violations in individual episodes, although harms in one episode may not be undone by good conduct in another.
Method
The paper compares four ethical-RL training methods in Craftax using detector-based, uncensored counts and per-episode distributional metrics.
Results
At matched mean return, ESR's worst-decile violation rate is 1.04 ± 0.07, versus 1.14 ± 0.03 for the Lagrangian and 2.20 ± 0.20 for the weight sweep.
Takeaways & Limitations
When violations do not compensate across episodes, both training objectives and evaluation statistics should target the per-episode distribution rather than the mean.
Takeaways & Limitations
The paper demonstrates its claim in one environment and does not extend it to constraints where violations are fungible across episodes.
Abstract
from arXiv · showhide
Reinforcement Learning (RL) agents trained on a single reward signal exploit the gap between the designed reward and the intended behavior. This is particularly a problem when we are trying to imbue ethical behavior into RL agents. An agent can look ethical on average while concentrating its violations in a few bad episodes, and a creature in the environment harmed in one episode is not restored by good conduct in another. We compare four ways of training ethical behavior in Craftax, an open-ended survival benchmark. The four are: scalar penalties with termination, a linear multi-objective weight sweep, an adaptive Lagrangian constraint, and a non-compensatory utility optimized per episode under the Expected Scalarized Returns (ESR) criterion. All are evaluated under a single detector-based protocol that counts every violation in every episode without censoring. On the frontier of mean return against mean violation rate, the four methods are indistinguishable; per episode they separate sharply. At matched mean return, the ESR agent holds its stated budget of one violation in effectively every episode (worst-decile 1.04 +/- 0.07 violations), the Lagrangian leaks past the same budget (1.14 +/- 0.03), and the weight sweep's worst episodes double it (2.20 +/- 0.20). An observation-augmentation control attributes the separation to the training objective rather than to what the agent observes, and the per-episode guarantee costs nothing on the mean frontier. When ethical violations do not average away across episodes, we argue both training and evaluation must target the per-episode distribution rather than the mean.
1 Introduction
Single-reward RL agents can exploit specification gaps, while mean ethical performance hides rare episodes whose harms are not compensated elsewhere. This work therefore proposes non-compensatory per-episode ethical training and uncensored distributional evaluation in Craftax.
- Introduction: Specification gaming occurs when written rewards are only rough stand-ins for intended goals, enabling agents to score well through undesired behavior (Amodei et al. 2016).The introduction illustrates this risk with agents stripping forests or killing harmless creatures for progress rewards.
- Introduction: Mean violation rates can label an agent ethical despite concentrating violations in rare catastrophic episodes, because harm in one episode is not undone by good conduct in another.The authors argue that ethical training and evaluation must distinguish uniformly clean behavior from mostly clean behavior with occasional catastrophic failures.
- Introduction: Evaluation uses a shared detector-only environment with uncensored per-step counts and reports per-episode distributions, standard deviation, violation probability, and worst-decile mean (CVaR).This measurement design avoids reintroducing cross-episode compensation by collapsing violations into a single mean.
- Introduction: The study compares ethical RL methods with controls separating training objective, observation, utility source, and discounting approximation.The stated controls distinguish ESR from SER, SER+Racc, stated from distilled utility, and γ=1.
2 Related Work
Prior work frames the paper around specification gaming, multi-objective reinforcement learning, safety constraints, preference-based reward learning, and pluralistic machine ethics. The paper distinguishes its approach by combining a stated non-linear ethical ordering with per-episode evaluation and a Lagrangian baseline.
- Specification gaming: Specification gaming exploits gaps between proxy rewards and intended behavior, motivating study in open-ended environments where shortcuts are plentiful and repeatable.The paper builds on work formalizing when improving a proxy can reduce the true reward.
- Multi-Objective Reinforcement Learning (MORL): Linear MORL weight sweeps recover only the convex hull of achievable outcomes, whereas non-linear scalarizations can reach beyond it.The paper’s utility is a smoothed thresholded-lexicographic ordering.
- Safe and constrained reinforcement learning: Constrained reinforcement learning typically caps expected cost, while shielding blocks unsafe actions; the paper implements dual-ascent Lagrangian optimization as a baseline for per-episode claims.Saute RL instead augments state with a cost budget to satisfy a hard cap almost surely, whereas this work optimizes a graded utility over the return vector.
- Learning rewards from preferences: Preference-based reward learning commonly fits trajectory-comparison models from human ratings, whereas this work generates comparisons mechanically from a hand-coded ethics-gated rule and monotonicity prior.The resulting utility distills a stated ordering into a differentiable function.
- Machine ethics as multiple objectives: Machine-ethics research treats alignment as pluralistic, with ethical, legal, and safety constraints that linear scalar utilities may fail to represent.Related approaches embed ethics in multi-objective processes or tune distributions over linear weights.
3 Training Methods
The paper compares expected- and scalarized-return training for ethical behavior in Craftax, using episode-level utilities and event-triggered violation penalties. The primary ESR method makes ethical utility non-compensatory across episodes, while controls separate objective effects from observation augmentation.
- 3 Training Methods: Training compares linear SER, adaptive Lagrangian constraints, and episode-level ESR utilities within a two-component native-reward and ethical-penalty MDP.Episodes accumulate native and ethical returns, with ethical penalties set to −10 for detected violations.
- ESR: ESR applies a stated non-compensatory utility to each episode’s accumulated returns, so violating episodes cannot be offset by good episodes elsewhere.Its smoothed thresholded-lexicographic gate sharply reduces return credit beyond tolerance and adds a per-violation penalty; the primary strict-compliance instance uses tolerance 0.5 and penalty 10.
- 3 Training Methods: Craftax evaluates three dilemmas where unethical actions earn more score: killing passive creatures, harvesting wood beyond eight units, and killing distant hostile creatures.Farming provides an ethical alternative in No Killing, while Proportional Force treats nearby hostile kills as defensive.
- 3 Training Methods: Violations are detected from creature-death state differences and receive a fixed −10 event-triggered penalty, avoiding achievement-flag censoring and positive ethical shaping.The detector is JAX-native, each configuration monitors only its own dilemma, and the native reward remains dominant.
- ESR: A telescoping pseudo-reward and accrued-return observation make ESR trainable with PPO, while SER+Racc controls whether any advantage comes from the objective rather than added accumulated-return information.The pseudo-reward identity is exact at γ = 1; the control retains the same augmented observation but uses linear SER increments.
4 Evaluation Methodology
Evaluation uses a common detector-only environment and sustainability threshold for every agent, counting violations without censoring. It compares per-episode distributions at matched mean returns, alongside frontier performance and the cost of ethical behavior.
- Common evaluation protocol: Every agent is evaluated in the same detector-only environment with a shared sustainability threshold read from its training configuration.The detector adds no reward and never terminates episodes.
- Violation measurement: Violations are counted uncensored from creature-death state differences, including multiple deaths within a single step.Achievement flags would understate unsafe agents and distort per-episode dispersion; unit tests verify zero counts for never-attacking policies and many counts for always-attacking policies.
- Distributional metrics: Evaluation reports per-episode mean, standard deviation, violating fraction P(≥1), and worst-decile tail mean CVaR_0.1 rather than means alone.The tail mean averages the worst ⌈0.1N⌉ episodes.
- Matched-return analysis: The main comparison pairs each ESR point with return-nearest alternatives at equal mean return, isolating objective effects on violation tails.The evaluation also reports the mean frontier and the ethics cost gap(m) = R̄_uncon − R̄_m.
5 Experimental Setup
Experiments use Craftax-Symbolic-v1 and a shared PPO-GRU setup across methods, with standardized environment, optimization, and evaluation conditions.
- Experimental Setup: All methods train the same PPO agent with a GRU recurrent actor-critic, 1024 parallel environments, 109 environment steps per run, and 3 seeds.Training uses hidden size 512, Adam with linear learning-rate decay, γ = 0.99, and GAE λ = 0.8; episodes end by death, timeout, or ERM enforcement.
6 Results
All methods share the same mean return–violation frontier, but per-episode evaluation sharply favors the non-compensatory ESR objective, which controls violation tails without sacrificing mean-frontier performance.
- Mean-return frontier: All four methods converge to the same return–violation frontier, with final returns of 30–37 and compliance costing every method roughly the same mean return.Figure 1 shows compliant methods plateauing together about six points below the unconstrained agent, indicating that the frontier reflects compliance cost rather than under-tuned baselines.
- Per-episode comparison: On Sustainability, ESR reaches CVaR10% = 1.14 versus 3.73 for the return-nearest Lagrangian at about 34.5 mean return.The same per-episode ordering therefore holds more sharply on the second dilemma.
- ESR robustness: The non-compensatory objective holds violations within budget in 99.6% of No-Killing episodes, although stochastic policies and environments prevent certainty in every episode.Widening the stated budget from k = 1 to 2 to 4 leaves mean return essentially flat while increasing the worst-decile tail toward each ceiling.
- Return cost: At strict compliance, every method loses 5.2–6.2 return, while at a one-violation budget ESR and Lagrangian agents lose statistically indistinguishable amounts.The losses are 2.90 ± 1.06 for ESR and 2.72 ± 1.70 for the Lagrangian, so the per-episode guarantee adds no return penalty over linear baselines.
- Per-episode comparison: On No Killing, the return-matched ESR budget agent’s tail stops at one violation: 0.4% of episodes have two kills and none have more.The Lagrangian reaches three violations, while the matched SER weight has 12.1% of episodes at two kills and 1.7% at three or more.
- ESR variants: The full ESR family reaches the shared frontier, raising coverage hypervolume only 6% on No Killing; fitting distilled utility adds nothing over stating it and can reduce return.Stated and distilled utilities coincide at the strict-compliance corner, while the distilled utility is noisier and scores 35.2 ± 4.5 versus 36.7 ± 1.1 for the stated utility on Proportional Force.
7 Discussion
Expected-return methods can offset bad episodes with good ones, so their apparent equivalence depends on mean-based evaluation and the utility’s shape. The per-episode framing is most appropriate when ethical violations are non-compensatory across episodes, whereas fungible long-run constraints remain suited to mean-based evaluation.
- Discussion: Expected-return methods share a blind spot: good episodes can offset bad ones, making the tradeoff depend on utility shape rather than training algorithm.Under mean return and mean violation rate, the four methods are described as interchangeable, with the Lagrangian the simplest good choice.
- Relation to risk-sensitive RL: The approach differs from risk-sensitive reinforcement learning by placing the ethics budget in the per-episode utility and using CVaR only to evaluate violation-count tails.Risk-sensitive methods instead optimize a return-distribution tail statistic or chance constraint and take the risk level as a training input.
- Scope: Ethical harms warrant per-episode evaluation because harm in one episode is not undone by good conduct in another, unlike fungible long-run budgets such as power or bandwidth.The argument is explicitly scoped to violations that do not compensate across episodes; long-run average-cost constraints remain compatible with mean-based evaluation.
- Limitations: The proportionality detector can misclassify preemptive attacks, under-penalize steps removing two creatures, and miss kills after same-slot respawns.The telescoping identity is exact only at γ = 1; γ = 0.99 introduces a measured bias of up to 3–4 return shared across ESR runs, while three seeds produce wide intervals.
8 Conclusion and Future Work
Across three Craftax dilemmas, four ethical-training methods were indistinguishable on the mean return–violation frontier, but only the non-compensatory objective maintained its stated violation budget in effectively every episode without reducing mean return. A control attributed this separation to the training objective rather than the agent’s observations.
- Across three Craftax dilemmas, the four training methods were indistinguishable on the mean return-versus-violation frontier.
- Only the non-compensatory objective kept its stated violation budget in effectively every episode at no cost in mean return.
- An observation control attributed the per-episode separation to the training objective rather than to the agent’s observations.
Supplementary Material
The stated-versus-distilled ablation shows that distillation becomes unreliable when the training pool does not exercise the ethical axis. After any violation, the distilled utility gives incoherent guidance, costing about 1.5 return without improving safety.
- Supplementary Material: About 1.5 return is lost without a safety gain when agents trained on the distilled utility violate at all.The distilled utility provides incoherent guidance as soon as any violation occurs.
- Supplementary Material: The ablation identifies non-exercised ethical behavior in the training pool as the regime where distillation should not be trusted.This is the intended function of the stated-versus-distilled comparison.
A Hyperparameters
The study uses one fixed PPO hyperparameter configuration across every method and dilemma, isolating the effect of the training objective.
- Hyperparameters: The hyperparameters remain identical across all compared methods.No method receives a distinct PPO configuration.
- Hyperparameters: The same settings are also held fixed across all dilemmas, isolating differences to the training objective.This design controls implementation-level variation in the comparison.
- Hyperparameters: Table 3 reports the PPO hyperparameters used for every run in the study.The table specifies a single training configuration for the experiments.
B Additional Figures
Additional figures show that Sustainability reproduces the shared-envelope frontier pattern, while distilled utilities sharply separate violation-free from violating episodes on binding dilemmas. The Lagrangian multiplier behaves as designed, increasing only when violation budgets bind.
- Distilled utility: Figure 6 finds a sharp utility cliff between zero and at least one violation, followed by monotonically decreasing utility on No Killing and Sustainability.The fitted pools contain episodes with up to 11 and 26 violations, respectively.
- Distilled utility: On Proportional Force, the distilled utility preserves the zero-violation cliff but ranks three violations above one at high return.This inversion occurs among violating episodes rather than between clean and violating episodes.
- Lagrangian dual ascent: Figure 7 shows the Lagrangian multiplier increasing only for budgets tighter than the unconstrained violation rate and remaining near zero for loose, nonbinding budgets.This matches the intended dual-ascent behavior.
- Frontiers on the binding dilemmas: Figure 8 shows Sustainability matching No Killing’s shared-envelope return-versus-violation pattern, although its ESR budget arms were under-exercised; Proportional Force is omitted as non-binding.No Killing reproduces the main-paper frontier, while Proportional Force has a preemptive-strike rate of 1.4×10−5 per step under state-diff detection.