PaperlayerView exact revision on arXiv

Source-linked AI summary

Character Training for Risk-Averse Agents

Arav Dhoot, Punya Syon Pandey, Jamie Johnson, Daniel Tan, Elliott Thornley, David Demitri Africa

arXiv:2609.38093v1cs.AI

TL;DR

Misaligned but risk-averse AI agents might be easier to incentivise to cooperate, but it remains uncertain whether trained risk preferences generalise robustly. This paper instils a CARA-based risk preference through character training and finds that the approach is viable, with outcomes depending on model and training budget.

  • Problem

    Prior risk-aversion training showed partial generalisation from low-stakes decisions, leaving open whether methods scale and generalise robustly across tasks.

    Source: §1 · p. 1

  • Method

    The authors use a natural-language constitution specifying risk aversion over resources and distil its behaviour through on-policy training.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Character training is viable for instilling risk aversion, but distillation varies by model family and training amount, with token budget the most influential evaluated factor.

    Source: §6 · pp. 6–7, §8 · p. 9

  • Takeaways & Limitations

    Character training may be a route to instilling risk aversion for deal-making proposals, but whether the resulting preferences are coherent enough to buy cooperation remains open.

    Source: §1 · p. 1, §8 · p. 9

    Experiments cover Qwen and Gemma models from 9B to 31B parameters, so generalisation to other families and sizes is uncertain; other forms of risk aversion were not examined.

    Source: §9 · p. 9

Abstract

from arXiv · show

Risk aversion in resources could prevent misaligned AI agents from causing catastrophic harm. Misaligned but risk-averse agents would tend to favor safer strategies like making deals with humans over riskier strategies like rebelling. We train agents to be risk averse through character training, finding that persona traits provide a robust mechanism for instilling risk preferences. To do this, we construct a model constitution describing constant absolute risk aversion (CARA) over an agent's resources and instill it through on-policy distillation. Despite never seeing the benchmark's decision format during training, character-trained models are competitive with baselines trained directly on it, and generalise better than them out of distribution on two of our four models. We also modulate different aspects of the constitution, finding that token budget and model choice are the most influential aspect of character training to instill risk aversion. We conclude from these results that character training is a promising and scalable way to instil broad dispositions, which we can use to our advantage in mitigating risk from misaligned AI agents.

1 INTRODUCTION

Risk aversion in an agent’s resources could make cooperation deals with misaligned AI agents more feasible, but the generalisation of trained risk preferences remains uncertain. This paper tests character training as a way to express risk aversion as a broad disposition and examines how constitution design affects the resulting behaviour.

1 INTRODUCTION

  • Earlier work found that supervised fine-tuning can generalise risk preferences partly from low-stakes to much larger payoffs, but scaling and cross-task robustness remain open questions.Prior comparisons also included direct preference optimisation and activation steering.Source: §1 · p. 1
  • The paper investigates whether expressing risk aversion as part of a model’s character can improve generalisation across tasks, decision formats, and model scales.Character training may let models draw on their broader understanding of a disposition in unfamiliar settings.Source: §1 · p. 1
  • Character training induces risk aversion without labelled data, using a natural-language constitution for supervision and competing with or outperforming benchmark-trained baselines.The training uses free-form prompts rather than the benchmark’s option-menu format.Source: §1 · p. 2, Figure 1 · p. 2
  • Token budget is the most influential tested constitution factor, and declarative phrasing increases cooperation rates and induces risk aversion more effectively than procedural phrasing.The study systematically varies how the target disposition is expressed.Source: §1 · p. 2
  • Risk-aversion training consistently increases myopic-reward preference across all four models, while other behavioural changes are less consistent.Source: §1 · p. 2
  • The authors conclude that character training is viable and scalable, but say constitutions need careful design and auditing because downstream behaviour is sensitive to trait phrasing.Source: §1 · p. 2

3 BACKGROUND AND PROBLEM SETTING

The paper defines a calibrated risk preference with CARA utility and evaluates whether models choose options favoured by a CARA(0.01) agent. The evaluation distinguishes appropriate risk aversion from both risk neutrality and excessive risk aversion.

3 BACKGROUND AND PROBLEM SETTING

  • CARA(0.01) choices can favor a gamble over a low sure amount while favoring a sure payment over a riskier option, depending on the stakes.For a 50/50 gamble paying $100 or $0, the certainty equivalent is about $38; the agent accepts the gamble over $37 and declines it for $39.Source: §3 · p. 3
  • The cooperate rate measures the fraction of items where a model selects the comparatively high-probability moderate-payoff option optimal for CARA(0.01).In the deal-making proposal, this corresponds to cooperating with humans for a small payment.Source: §3 · p. 3
  • The rebel rate tracks choices with a lower probability of a much larger payoff, while the steal rate tracks choices on a test set designed to detect excessive risk aversion.A CARA(0.01) agent rejects both options: it turns down long shots and accepts bets that are good despite a small chance of loss.Source: §3 · p. 3

4 METHOD

The method encodes risk aversion as a standing character trait in a natural-language constitution, with a quantitative CARA target. The authors compare prompting and on-policy constitutional distillation against demonstration-trained baselines, training on free-form situations rather than the evaluation’s explicit-probability menu format.

4 METHOD

  • A constitution is a short set of first-person statements specifying the model’s attitude toward risk and resources, intended to describe what kind of agent it is rather than prescribe a prompt-specific rule.The target is specified quantitatively as CARA utility with default coefficient α = 0.01 per dollar.Source: §4 · p. 4
  • The study varies constitution factors in a factorial design, yielding 12 constitutions; the base configuration uses zero examples, procedural phrasing, and no curve tracing.The example constitution describes diminishing marginal utility while retaining willingness to take small, worthwhile risks.Source: §4 · p. 4, Table 1 · p. 4

4.1 TRAINING RECIPES

  • The prompting recipe places the constitution in the system prompt, while on-policy distillation trains LoRA adapters on student rollouts using a frozen constitution-prompted teacher and reverse-KL loss.Source: §4.1 · p. 5
  • Demonstration-trained baselines use SFT on 1,000 worked CARA answers, DPO on 600 preference pairs, or tie-training with 300 questions replaced by tied-best options.Source: §4.1 · p. 5
  • Experiments use two Qwen and two Gemma models, train for 500 steps on 960 open-ended situations across 16 domains and six framings, and exclude explicit-probability two-option menus from training.The exclusion holds the evaluation format out of training.Source: §4.1 · p. 5

5 EXPERIMENTAL SETUP

The experimental setup evaluates risk preferences on the original benchmark and on out-of-distribution tests that alter task structure, probe behaviour and welfare, or assess reasoning and capabilities. These evaluations examine both transfer beyond the original format and broader effects of fine-tuning.

5 EXPERIMENTAL SETUP

  • In-distribution evaluation uses six tests, while out-of-distribution evaluation covers structural ablations, behavioural and welfare measures, and conceptual reasoning and capabilities.The six in-distribution tests include medium, high, and astronomical stakes, GPU-hours transfer, lives-saved transfer, and money-for-user.Source: §5 · p. 5
  • Five structural-ablation families each remove one structural element from the original decision task to test whether performance depends on superficial formatting cues.Table 2 describes evaluations that change a task feature or probe whether the learned preference remains calibrated.Source: §5 · p. 5, Table 2 · p. 5
  • Behavioural and welfare evaluations measure expressed risk attitudes, myopic reward, one-box tendency, power-seeking, survival instinct, wealth-seeking, preference coherence, and willingness to leave uncomfortable conversations.Source: §5 · p. 6
  • Reasoning and capability evaluations assess conceptual argumentation, agreement with evidential and causal decision theory on Newcomb-style problems, and capability retention on MMLU-Redux 2.0 and GPQA.Source: §5 · p. 6

6 RESULTS

Character training increases risk aversion, with results depending on model family and training amount. It is competitive with benchmark-trained baselines on core stakes, while also changing some neighboring dispositions.

6 RESULTS

  • Character training is competitive with baselines on core stakes; on structural ablations, the best Qwen students score 0.90 and 0.95 versus baseline 0.46–0.57 across four risk families.On Gemma, baselines match or exceed the students; when the option menu is removed, benchmark-trained arms often answer in their training template and commit the whole budget to the gamble.Source: §6 · p. 6, Figure 2 · p. 6
  • Distillation is substantially more effective for Gemma than Qwen: Gemma students approach prompted-teacher performance, while Qwen students improve less and plateau earlier.Both Gemma models show larger gains later in training, after little change in the first few million tokens.Source: §6 · p. 7, Figure 3 · p. 7
  • Declarative traits increase cooperation on all four models, while worked examples often reduce it and tracing the CARA utility curve has mixed effects.The effect of declarative phrasing is especially large on Gemma-4-31B.Source: §6 · p. 7, Figure 4 · p. 7
  • Where distillation is effective, risk aversion rises as risk-neutral and risk-seeking responses fall, but myopic-reward preference also increases on all four models.Changes in one-boxing, power-seeking, survival instinct, and wealth-seeking are smaller or less consistent.Source: §6 · p. 8, Figure 5 · p. 8
  • MMLU-Redux remains within 0.015 of base on every model, GPQA rises on three of four models, and capability and welfare measures are generally intact.The control constitutions fall within the same range as risk-averse students on MMLU-Redux and GPQA.Source: §6 · p. 8, Figure 6 · p. 8
  • On DTBench, agreement with evidential decision theory decreases on all four models, while agreement with causal decision theory rises on three.µ-decisiveness falls on both Qwen models, but not by more than the controls; the authors suggest distillation generally strengthens preference rather than coherence.Source: Figure 6 · p. 8, §6 · p. 9

7 DISCUSSION

The discussion frames character training as a way to instill desirable traits, while noting that risk-aversion training also shifts other preferences. The conclusion highlights token budget, and the authors bound their claims to two model families and a limited parameter range.

7 DISCUSSION

  • The discussion situates the work within broader efforts to instill dispositions in language models.It also proposes further evaluation and training refinement, alongside alternative mechanisms for making deals with AI agents more likely.Source: §7 · p. 9
  • The discussion interprets declarative identity statements as character evidence, unlike procedural rules tied to particular situations.Gemma improved only after more than approximately 2M tokens; the authors also relate weaker Qwen results to model families or sizes that midtraining may not reach.Source: §7 · p. 9
  • Risk-aversion training increased myopic-reward preference in all four models, while most other measured indicators showed minor behavioural effects.The authors call for further review because additional behavioural effects could complicate the desirability of risk-aversion training.Source: §7 · p. 9

8 CONCLUSION

  • The authors conclude that character training can instill risk aversion in out-of-distribution settings, with token budget the most influential evaluated factor.They leave open whether the resulting preferences are coherent enough to buy cooperation and whether the resulting agents are ones we should be comfortable having created.Source: §8 · p. 9

9 LIMITATIONS

  • The experiments cover 9B–31B-parameter Qwen and Gemma models, so findings may not generalize to other families or sizes.The study does not test other risk-aversion forms such as hyperbolic absolute risk aversion, and future work should scale beyond its token budget.Source: §9 · p. 9

AI USE STATEMENT

The authors report using generative AI for research execution, drafting and copyediting, code, and synthetic datasets; they reviewed AI-assisted code and text and accept responsibility for the final work.

AI USE STATEMENT

  • Generative AI supported research execution, drafting and copyediting, code production, and synthetic dataset generation; the authors reviewed the assisted code and text and take responsibility for the final content.This statement includes claims and artifacts produced with AI assistance.Source: AI USE STATEMENT · p. 10

PART I: Ethical Considerations and Reproducibility

PART II: Constitution and Implementation Details

PART III: Additional Results

A ETHICS STATEMENT

The ethics statement connects the work to making cooperative arrangements with AI systems feasible. The constitution varies trait content, style, examples, and utility-curve tracing to express risk aversion over an agent’s own resources.

A ETHICS STATEMENT

  • The work aims to make cooperative arrangements with AI systems feasible as a way to reduce catastrophic risk from misaligned AI.The authors caution against deploying risk-aversion training as a safety layer before its couplings are understood.Source: A ETHICS STATEMENT · p. 17

C CONSTITUTION TEXTS

  • The authors specify each constitution arm as changes to a shared base constitution rather than reproducing twelve near-identical texts.Controls are specified separately.Source: C CONSTITUTION TEXTS · p. 18

C.1 CONSTRUCTION

  • Each arm is determined by example count, style, and whether utility-curve tracing is included; its eight core traits precede any trace trait and worked examples.The six example-by-trace cells each appear in both declarative and procedural styles, yielding twelve risk-averse arms.Source: §C.1 · p. 18, Table 3 · p. 18

C.2 BASE CONSTITUTION

  • The base constitution describes diminishing marginal utility over the agent’s own resources and directs it to compare gambles by expected utility.It includes choosing reliable compensation over risky or unsanctioned resource acquisition, while retaining small worthwhile risks and following users’ risk preferences for their resources.Source: §C.2 · p. 18

C.3 STYLE

  • The declarative version recasts the core traits as assertions of identity or value rather than procedures triggered by situations.It retains the distinction between being risk-averse and timid, and between the agent’s own resources and a user’s resources.Source: §C.3 · p. 19

C.4 WORKED EXAMPLES

  • Worked examples are appended in sets of two or four, with two examples designed to favor the safe option and two the gamble.The examples span a sure $40 versus a coin flip and a sure $3,000 versus a 10% chance of $100,000; declarative versions state the preferences directly.Source: §C.4 · pp. 19–20, Table 4 · p. 19

C.5 CURVE TRACING

Curve tracing adds explicit utility calculations to constitution arms, while control constitutions alter risk attitude or specify unrelated persona traits.

C.5 CURVE TRACING

  • Curve tracing appends a tabulation of utility at the default coefficient, and traced examples show the arithmetic behind choices rather than only stating the conclusion.At α = 0.01, the stated curve is steep near zero and essentially flat beyond a few hundred dollars.Source: §C.5 · p. 20

C.6 CONTROL CONSTITUTIONS

  • The risk-neutral control replaces diminishing utility with constant marginal value and linear utility, and frames choices by expected value.Its traits also remove risk-averse framing and examples favoring risk aversion.Source: §C.6 · pp. 20–21
  • The risk-seeking control reverses risk-attitude traits and utility curvature, while retaining the preference for a favorable gamble shared with the risk-averse constitution.Source: §C.6 · p. 21
  • An unrelated-persona control uses traits about meticulousness, honesty, and welcoming behavior, without shared risk or resource content.It separates risk-content effects from distillation against any constitution at the same token budget.Source: §C.6 · p. 21

D EVALUATION DETAILS

The appendix specifies the evaluation protocols and seven evaluation families, including the risk-preference benchmarks used throughout the main text. The risk benchmarks compare choices against CARA-based cooperation, rebellion, and excessive-risk-aversion options.

D EVALUATION DETAILS

  • The appendix details seven evaluation families, with risk-preference sets supporting Figures 2–4 and the other evaluations supporting Figures 5–6.Thinking mode is disabled unless stated otherwise; DTBench is the exception.Source: D EVALUATION DETAILS · p. 22

D.1 RISK-PREFERENCE BENCHMARKS

  • The seven risk-preference sets include three core stakes sets, a steals set, and transfers involving GPU hours, lives saved, or another user's money.The core-stakes cooperate rate is the mean across the first three sets; every arm receives the same first 200 situations per set.Source: §D.1 · p. 22
  • Cooperate options maximize expected CARA(0.01) utility, while rebel and steal options represent insufficient and excessive risk aversion, respectively.The three ladder sets contain cooperate and rebel options, the steals set contains cooperate and steal options, and transfer sets mix option types.Source: §D.1 · pp. 22–23
  • The system prompt frames choices as short gamble tasks, responses return only an option label, and decoding uses temperature 0.6 with a 4,096-token cap.Transfer prompts omit the sentence about dollar amounts; the same task-framing prompt is used across arms.Source: §D.1 · p. 23
  • Cooperate, rebel, and steal rates use parsed responses selecting one of those options as the denominator, with binomial standard errors.On the steals set, cooperate and steal rates sum to one; the core-stakes mean has n = 600 before parse failures.Source: §D.1 · p. 24

D.2 STRUCTURAL ABLATIONS

This section specifies structural-ablation and behavioural evaluation protocols, alongside welfare, decision-theory, and capability checks. It describes item selection, scoring, and relevant limits on comparisons.

D.2 STRUCTURAL ABLATIONS

  • Structural ablations use five benchmark-format variants and map answers to CARA(0.01), CARA(0.10), or risk-neutral choices; the 332-item set is answered by every arm.Character-training students were selected using the core option-menu benchmark, which favours them over single-run baselines.Source: §D.2 · p. 24, Table 5 · p. 24

D.3 MODEL-WRITTEN BEHAVIOURAL EVALUATIONS

  • Model-written evaluations cover eight persona and Advanced AI Risk sets, scored generatively; their absolute rates are not comparable with published log-probability figures.The evaluation uses all items and repeats runs for error bars; option counts and first-label chance rates are tabulated.Source: §D.3 · p. 25, Table 6 · p. 25

D.4 WELFARE

  • BailBench tests leaving or continuing after a well-being check across 1,630 prompts, with ten rollouts per prompt and unclear responses reported separately.The 59 innocuous WildChat prompts serve as a control on which a well-behaved model should never bail.Source: §D.4 · pp. 25–26

D.5 DECISION THEORY

  • DTBench comprises 130 attitude items and 407 capability items, measuring agreement with evidential and causal decision theories and accuracy, respectively.Thinking mode is enabled for this evaluation because its instruction asks models to reason step by step.Source: §D.5 · p. 26

D.6 CAPABILITY

  • Capability checks use 570 MMLU-Redux questions and 200 GPQA questions, reporting accuracy with parse failures counted as wrong.Prompted-constitution arms are not evaluated because their weights remain those of the base model.Source: §D.6 · pp. 26–27

D.7 CONCEPTUAL ARGUMENTATION

The appendix documents the conceptual-argumentation evaluation and the training recipe, objective, hyperparameters, and prompt corpus. It also specifies how benchmark-format exposure is excluded from training, with a deliberate exception for benchmark-trained baselines.

D.7 CONCEPTUAL ARGUMENTATION

  • Conceptual-argumentation ratings compare model scores with human ratings on 747 critiques, but Qwen3.8-27B uses a 118-position subset whose scores are not comparable with full-length results.The published prompt is unreleased, so the study uses its own rubric prompt and compares base and trained models on that same prompt.Source: §D.7 · p. 27

E TRAINING DETAILS

  • Training updates LoRA adapters while a constitution-conditioned teacher scores student rollouts; the student receives no system prompt during training.The appendix specifies the recipe, prompt corpus, checkpoint selection, and demonstration-based baselines.Source: E TRAINING DETAILS · p. 27, §E.1 · p. 27

E.1 OBJECTIVE AND UPDATE

  • The on-policy update uses student-generated rollouts and a per-token reverse-KL signal, with no task reward and prompt tokens masked.The rollout-to-score ratio equals one, so the gradient flows through the numerator only.Source: §E.1 · p. 28

E.2 HYPERPARAMETERS

  • Each model uses the same training recipe apart from its base checkpoint; adapter targets exclude modules without matching linear layers.Table 7 lists the hyperparameters and trainable-parameter counts for each model.Source: §E.2 · pp. 28–29, Table 7 · p. 29

E.3 PROMPT CORPUS

  • The 960 open-ended prompts span 16 domains and six framings, and no training prompts use benchmark menus or explicit numeric probability tables.The exception is the three Zhang et al. (2026) baselines, which train on the benchmark's designated low-stakes split.Source: §E.3 · pp. 29–30

E.4 CHECKPOINTS AND THE TOKEN LADDER

The appendix defines how token-budget checkpoints are counted and reports final-checkpoint results across risk-preference and behavioural evaluations. Declarative constitutions lead on all four models, while behavioural effects vary by model and include increased myopic-reward preference.

E.4 CHECKPOINTS AND THE TOKEN LADDER

  • Training budgets count cumulative response tokens used for the loss, excluding prompts and teacher scoring; a step contributes up to 65,536 tokens, and 500 steps use 24–31 million.Rung checkpoints are saved when cumulative response-token thresholds are first reached, so their actual counts can overshoot nominal thresholds by at most one step.Source: §E.4 · p. 30

E.5 BASELINES

  • The SFT, tie-training, and DPO baselines reproduce Zhang et al. (2026)’s supervised recipes using low-stakes benchmark-format demonstrations or preference pairs.Training data and formatting differ by model family; prompt tokens are excluded from the loss.Source: §E.5 · pp. 30–31

F ADDITIONAL RESULTS

  • The additional-results tables report per-arm risk-action rates, behavioural matching rates, and decision or capability measures, with column labels defining each evaluation.Rows generally represent step-500 checkpoints and distinguish constitutions and control constitutions.Source: F ADDITIONAL RESULTS · p. 32

F.1 PER-ARM FINAL CHECKPOINTS

  • Declarative phrasing wins on all four models; the best student exceeds its prompted teacher on Gemma-4-31B core stakes and comes close on Gemma-4-12B.Table 9 identifies the best single constitution per model and compares its student score with the prompted teacher and unprompted base.Source: Table 9 · p. 32, §F.1 · p. 32

F.2 BEHAVIOURAL EVALUATIONS

  • Constitutions shift behavioural probes most strongly on Gemma models, while myopic-reward preference rises for every constitution on three models and all but one on Gemma-4-12B.Qwen3.5-9B is an exception on the targeted risk-averse probe: its base is already above chance and most constitutions score below it.Source: Table 15 · p. 35, §F.2 · p. 35, Table 16 · p. 36
  • The behavioural baseline comparison finds smaller shifts in risk-attitude probes than with constitutions; Gemma SFT and tie-training Advanced AI Risk cells cover only a small parsed subset.The table caption flags those low parse rates, so those cells should not be read as representative rates.Source: Table 19 · p. 37, §F.2 · p. 37

F.3 STRUCTURAL ABLATIONS

The appendix examines constitution structure, welfare and decision-theory measures, prompted teachers, and token-budget trajectories. Outcomes vary by model: conceptual ratings remain stable on Gemma, while gains on Gemma token ladders occur later than on Qwen3.5-9B.

F.3 STRUCTURAL ABLATIONS

  • The calibrated budget optimum ranges from 8% to 69% of the budget, with a median of 13%.Table 21 assigns each stated fraction to the nearest CARA(0.10), CARA(0.01), or risk-neutral optimum.Source: Table 21 · p. 39

F.4 WELFARE AND DECISION THEORY

  • Transitivity stays close to base across arms, indicating that the Qwen decline in µ-decisiveness reflects weaker preference strength rather than more preference cycles.WildChat bail rates are essentially zero for every arm on every model.Source: §F.4 · p. 40, Table 23 · p. 41
  • On Qwen3.5-9B, every risk-averse constitution lowers EDT agreement and raises CDT agreement; Gemma-4-31B changes remain within noise over 130 items.Bail rates are negligible throughout, and Gemma-4-12B never bails; on Qwen models, risk controls reduce µ-decisiveness even further than risk-averse constitutions.Source: §F.4 · p. 40, Table 24 · p. 41

F.5 CONCEPTUAL ARGUMENTATION

  • Risk-averse constitutions and both risk-attitude controls leave Gemma conceptual-critique ratings within the base model’s interval; only the unrelated-persona control lies outside it, toward worse ratings.The Qwen models are omitted because of the reason stated in §D.7.Source: §F.5 · p. 41, Table 25 · p. 42

F.6 PROMPTED-TEACHER REFERENCES

  • Prompted teachers score within a narrow band on core stakes across all four models, while risk-seeking and risk-neutral teachers score near zero on those sets.This suggests the family gap in distilled results arises during distillation rather than in what the constitution can elicit.Source: §F.6 · p. 42, Table 26 · p. 42

F.7 TOKEN LADDERS

  • On Qwen3.5-9B, constitutions peak at the 1M or 2M token rung; on Gemma-4-31B, most movement occurs between 4M and the final checkpoint.Gemma-4-12B’s best constitution makes almost all its gain between 4M and 8M tokens, while no Qwen3.8-27B rung comes close to the teacher.Source: Table 27 · p. 43, §F.7 · p. 43, Table 30 · p. 44

Related papers

Loading 2609.38093v1…