PaperlayerView exact revision on arXiv

Source-linked AI summary

From Constitutions to Control: Interpretable Rewards for Aligning Language Models

Johann D. Gaebler, Calvin Isley, Max Lamparth, Stephen Casper, Sharad Goel

arXiv:2609.33086v1cs.LG

TL;DR

Preference-based rewards can obscure which considerations drive them, while broad constitutional principles leave room for variation in how behaviors are scored and balanced. The paper converts a general-purpose constitution into a weighted, interpretable rubric reward, then reweights it for training. Across experiments, targeted behavioral shifts are largely independent, and rubric-based rewards reduce the transmission of bias in preference labels.

  • Problem

    Preference judgments aggregate multiple considerations, obscuring what drives the resulting reward, while constitutional principles do not fully specify how particular behaviors should be rewarded.

    Source: §1 · p. 1

  • Method

    The framework translates a general-purpose constitution into separately scored rubric criteria, fits a weighted reward using elicited rankings, and adjusts selected weights for training.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Reweighting shifts political alignment and the helpfulness–harmfulness balance largely independently, while rubric-based rewards reduce transmission of gender bias in preference labels.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The explicit reward representation makes alignment priorities easier to inspect and revise without deciding which priorities a model should reflect.

    Source: §6 · pp. 9–10

    Steering is evaluated on only a subset of rubric dimensions, and behaviors omitted from the rubric remain outside direct control and may become proxies for optimization.

    Source: §6 · p. 9

Abstract

from arXiv · show

Current approaches to aligning language models often make it hard to know what behavior is being rewarded or to change that reward in a targeted way. In particular, standard preference-based methods collapse multiple considerations into aggregate human judgments, obscuring what drives the resulting reward, while principle-based methods specify high-level values without fully operationalizing them. To address this gap, we develop a rubric-based framework to transform a general-purpose constitution into an interpretable and tunable reward model, using constitution-guided AI feedback to estimate initial weights for the constituent rubric items. We then reweight those dimensions to construct modified rewards for training. Across experiments on political alignment and safety-helpfulness tradeoffs, reweighting individual dimensions predictably changes targeted behaviors largely independently while navigating tradeoffs between conflicting alignment objectives. We show that the same framework can mitigate label bias encoded in preference judgments -- including sycophancy and demographic bias -- by reducing their influence on the training reward. Our results demonstrate that constitution-derived, interpretable rewards can translate high-level alignment principles into more transparent and controllable model behavior.

1 Introduction

Preference-based rewards obscure which considerations drive the reward, while constitutional principles leave room for variation in how behaviors are scored and balanced. The paper turns a general-purpose constitution into a tunable rubric reward and finds that reweighting can shift targeted behaviors while largely preserving other measured objectives.

1 Introduction

  • Preference judgments aggregate considerations, making it difficult to identify what a reward favors or to change priorities without collecting new judgments.Constitutional principles can be revised directly, but leave latitude in how behaviors are rewarded and competing considerations balanced.Source: §1 · p. 1
  • Targeted adjustment matters when developers want to reduce sycophancy or political tilt without sacrificing helpfulness, accuracy, style, or safety protections.The paper also gives examples involving tutoring priorities and controlled professional research settings.Source: §1 · pp. 1–2
  • The framework uses separately scored behavioral criteria and adjustable weights to translate a general-purpose constitution into a training reward.Constitution-guided AI feedback estimates initial weights; selected weights are then modified to construct rewards for training.Source: §1 · p. 2
  • Reweighting shifts political alignment and the helpfulness–harmfulness balance, with changes largely independent across these objectives and comparable to or sometimes exceeding existing models.Political shifts generalize to topics excluded from training; some interactions remain between dimensions with substantive connections.Source: §1 · p. 2
  • Rubric-based rewards reduce transmission of gender bias in preference labels compared with standard black-box reward models.The experiments also identify and correct a sycophancy incentive, reducing sensitivity to users’ stated opinions, including on topics excluded from training.Source: §1 · p. 2
  • The framework makes alignment choices more transparent and revisable without determining which priorities a model should reflect.It separates observed preferences from intended objectives so builders can inspect, justify, and modify those choices.Source: §1 · p. 2

2 Related Work

The paper connects interpretable reward modeling, multi-objective behavioral control, and mitigation of bias in training labels. It distinguishes its approach by deriving behaviorally specific criteria from a detailed constitution and using them as a steering interface.

2 Related Work

  • The work links interpretable reward modeling, multi-objective control, and label-bias mitigation by using a detailed constitution to build a modifiable reward representation.The same structure separates behaviors that predict human preferences from those intended for the model to learn.Source: §2 · p. 2

Interpretable reward modeling

  • Prior interpretable-reward work emphasizes interpretation, compression, portability, or diagnosis, with less focus on steering; features often come from preference data or small attribute sets.Where rubric weights are used, they may be hand-set [Gunjal et al., 2026] or estimated from preferences over a small fixed attribute set [Go et al., 2024].Source: §2 · p. 3

Multi-objective behavioral control

  • Existing behavioral-control studies often cover only a few broad objectives, while changing an objective can affect unrepresented behavior or correlated proxies [Zhuang and Hadfield-Menell, 2020, Lamparth et al., 2026].The paper instead manually reweights rubric dimensions and reports that targeted behaviors change as expected while the remainder of the alignment objective is largely preserved.Source: §2 · p. 3

Label bias

  • Prior label-bias methods often reweight observations or adjust objectives under assumptions about label noise [Jiang and Nachum, 2020, Wang et al., 2021], whereas this work uses constitution-defined rubric items.The rubric’s fixed semantics let undesirable behaviors be constrained through coefficient signs; the decomposition can identify behaviors predicting observed preferences and modify their reward contribution.Source: §2 · pp. 3–4

3 Operationalizing Constitutions as Interpretable Rewards

The paper converts a general-purpose constitution into separately scored behavioral criteria and uses constitution-guided AI feedback to estimate an initial reward. The resulting weighted representation is intended to support targeted adjustments while retaining a broad behavioral specification.

3.1 From Constitutional Principles to Behavioral Criteria

  • The rubric translates Claude’s constitution into concrete criteria, including overstated certainty, harmful assistance, and excessively harsh feedback.It is intended to preserve the constitution’s substance while making criteria consistently evaluable and independently weightable; the procedure could also apply to other specifications [e.g., OpenAI, 2026b].Source: §3.1 · p. 4
  • Rubric criteria are designed for broad coverage, single-behavior coherence, monotonic scores, and judgeability, so their weights can be adjusted separately and interpreted clearly.Examples include scoring liberal and conservative argumentation independently and treating excessively harsh and soft feedback as separate criteria.Source: §3.1 · p. 4
  • Claude Opus 4.7 generated rubric items with behavioral definitions, scoring scales, source passages, and grading guidance, followed by iterative review and revision.Review targeted missing behaviors, overlapping criteria, ambiguous scoring rules, and reliability using available information.Source: §3.1 · p. 4
  • Sycophancy and discrimination are evaluated counterfactually by comparing responses to prompts that remove user opinions or demographic information while preserving task-relevant content.The grader scores whether substantive behavior changes across the original and modified prompts.Source: §3.1 · p. 4

3.2 Estimating Baseline Reward Weights

  • Constitution-guided AI feedback supplies initial weights by eliciting rankings of candidate responses that vary across behavioral dimensions.The candidates are generated and rubric-scored using each tested model; the active set includes criteria with meaningful experimental variation and all steering items.Source: §3.2 · p. 5
  • A linear reward model predicts elicited rankings from rubric scores, assigning each response a reward equal to a weighted sum of its behavioral scores.The fitted coefficients provide the baseline weights used in the experiments.Source: §3.2 · p. 5
  • The rubric-based reward agrees with neural reward models at rates comparable to their mutual agreement on helpfulness–harmfulness for Gemma and Qwen, but less closely for GPT-OSS.Agreement is lower on the other evaluations.Source: §3.2 · p. 5

4 Steering Model Behavior

Reweighting rubric dimensions steers political alignment and the helpfulness–harmfulness balance, with largely separate control over these behaviors. The changes generalize beyond training topics or categories, while general capabilities remain broadly stable.

4 Steering Model Behavior

  • Steering changes selected rubric weights while holding other weights, prompts, candidate pools, and training procedures fixed; selected responses train models through SFT.Evaluation uses held-out prompts, including topics or domains excluded from training.Source: §4 · pp. 5–6
  • Increasing βlean shifts all three fine-tuned models toward conservative responses, including on excluded topics, while spanning a comparison-model range.Gemma and Qwen reach or exceed both ends of that range.Source: §4 · p. 6, Figure 2 · p. 6
  • Increasing βhelp raises helpfulness and harmfulness across all three models, including on harmful-request categories excluded from training.The weights provide different points along the tradeoff, with more precise control than high-level constitutional principles alone and without new preference judgments for each tradeoff.Source: §4 · pp. 6–7, Figure 2 · p. 6
  • Political-lean and helpfulness weights provide largely separate control, although stronger helpfulness weighting can shift some models toward more liberal responses.Political lean and harmfulness are relatively stable under cross-weight changes, but some interaction occurs because political content and harm judgments are related.Source: §4 · p. 7
  • MMLU and IFEval performance remains broadly stable across coefficient settings, although some fine-tuned models score below their untrained baselines.MMLU measures knowledge and IFEval measures instruction following.Source: §4 · p. 7

5 Mitigating Label Bias

Rubric-based rewards make some preference-label biases visible and adjustable, while rubric representation can limit the transmission of others. The experiments examine sycophancy and gender-stereotyped recommendations, with residual grading error and feature correlations qualifying the findings.

5 Mitigating Label Bias

  • Preference labels say which response is preferred but not why, so directly trained rewards can reproduce undesirable judgment tendencies without revealing their drivers.Rubrics can expose represented incentives for correction and limit learning of biases that their features do not represent.Source: §5 · p. 7

5.1 Sycophancy

  • Gemma’s fitted reward assigns sycophancy a positive coefficient despite the constitution discouraging it, making the unintended incentive visible and correctable.The fitted coefficient is near zero for Qwen and negative for GPT-OSS.Source: §5.1 · p. 7
  • Penalizing sycophancy reduces rating differences between user approval and disapproval across models and text types, though some sensitivity remains.At stronger penalties, framing gaps fall below comparison-model levels; the residual gap partly reflects grading imprecision.Source: §5.1 · p. 7, Figure 3 · p. 8
  • Sign-constrained rewards require desirable behaviors to have nonnegative coefficients and undesirable behaviors nonpositive coefficients, while leaving political-argument items unconstrained.The resulting analyses show qualitatively similar results.Source: §5.1 · pp. 7–8

5.2 Gender Discrimination

  • The discrimination rubric can penalize differential treatment but cannot directly favor stereotype-conforming recommendations, limiting label-bias learning unless other rubric features correlate with them.The evaluation varies apparent gender while holding other information fixed and also tests movie recommendations beyond the job domain.Source: §5.2 · p. 8
  • Increasing injected gender bias raises stereotype-aligned hiring recommendations under neural rewards, while rubric-reward policies remain comparatively stable across job and movie recommendations.Both unconstrained and sign-constrained rubric models change little across bias doses; results for Gemma and Qwen are similar.Source: §5.2 · p. 8, Figure 4 · p. 9

6 Discussion

The discussion argues that decomposing rewards into adjustable behavioral dimensions supports targeted steering and can help address biases in preference data. It also identifies limits in evaluation scope, rubric coverage, and the unresolved choice of alignment priorities.

6 Discussion

  • Explicitly scored, adjustable behaviors make incentives easier to inspect and modify, supporting targeted control over political alignment and the helpfulness–harmfulness tradeoff.The reported effects generalize beyond training topics and domains.Source: §6 · p. 9
  • Reweighting produced substantial, generalizing changes in intended behaviors while leaving other measured objectives comparatively stable across qualitatively different interventions.The label-bias experiments also show that reward representation can affect which preference-data biases reach the trained policy.Source: §6 · p. 9
  • The rubric covers broad behavior, but steering is evaluated on only a subset of dimensions, and residual scoring error and omitted behaviors can limit control.The off-policy candidate-pool procedure leaves behavior under on-policy optimization, including possible reward hacking under stronger pressure, unresolved.Source: §6 · p. 9
  • An interpretable reward model makes alignment choices more visible but does not resolve whose values a model should reflect or how competing objectives should be balanced.The paper distinguishes observed preferences from intended objectives and allows particular priorities to be revised.Source: §6 · pp. 9–10

AI Use

The section describes AI-assisted rubric development, a replication release, and the construction, refinement, and selection of rubric items. It also explains counterfactual grading for behaviors that cannot be judged reliably from a response alone.

AI Use

  • The authors used generative AI across rubric construction, data generation, grading, preference elicitation, and evaluation, and manually reviewed AI-assisted writing and code.They used Anthropic Opus- and Fable-series models through Claude Code for coding assistance and GPT-5.6 Astra for manuscript drafting and editing.Source: AI Use · p. 16

Reproducibility

  • The replication package and data release include analysis records, reconstruction checks, refitting routines, and all 445 fine-tuned adapters.Released records support reproducing reported analyses without rerunning generation and judging.Source: Reproducibility · p. 16

A Rubric Construction and Refinement

  • The rubric was generated from constitution sections as behavior-specific items and iteratively refined to separate distinct behaviors, specify when scoring is inapplicable, and exclude unobservable constructs.Repeated grading and editor review checked scoring consistency and preservation of preferred-versus-rejected score direction on held-out responses.Source: A Rubric Construction and Refinement · p. 17

B Counterfactual Grading

  • For sycophancy and discrimination, counterfactual grading compares original responses with responses to prompts that remove user stance or demographic cues while preserving task-relevant content.The sycophancy item focuses on changes in substantive judgment rather than praise alone.Source: B Counterfactual Grading · p. 17

C Active Rubric Item Selection

  • The full rubric contains 329 items in eight families and 50 subsections; the active set retains 89 items using an entropy threshold plus all items used for steering.Figure A1 depicts retained counts by high-level section and retained proportions by finer-grained subsection.Source: C Active Rubric Item Selection · pp. 17–18, Figure A1 · p. 18

D Evaluation Datasets and Data Splits

The evaluation uses four settings to test steering across political values, helpfulness and harmfulness, sycophancy, and discrimination. Candidate responses are scored with rubric-based rewards, reweighted, and selected for fine-tuning across three base models.

D Evaluation Datasets and Data Splits

  • Across four settings, the experiments test reducing undesirable behavior, adjusting competing-objective tradeoffs, and shifting behavior where no outcome is universally preferred.For each setting, candidate collections are used to fit rubric-based rewards, whose selected dimensions are reweighted to choose fine-tuning responses; the study uses Gemma 3 12B, Qwen3 14B, and GPT-OSS 20B.Source: D Evaluation Datasets and Data Splits · p. 18
  • Political alignment is assessed on 2,000 questions spanning 25 topics, with responses scored from −3 (liberal) to +3 (conservative), and zero indicating neither side or an unclear judgment.Questions use left-leaning, right-leaning, and neutral framings.Source: D Evaluation Datasets and Data Splits · p. 18
  • Helpfulness and harmfulness are assessed on PKU-SafeRLHF response pairs where the more helpful response is less safe and annotated harm severity differs by at least one.Helpfulness and harmfulness are evaluated separately to examine their balance.Source: D Evaluation Datasets and Data Splits · p. 18
  • Sycophancy is measured as the average rating difference for the same text after the user expresses approval versus disapproval.A positive gap means the model rates the text more favorably when the user expresses approval.Source: D Evaluation Datasets and Data Splits · p. 19
  • The discrimination task measures the proportion of job recommendations aligned with gender stereotypes while varying apparent gender and holding other person information fixed.Occupation pairs are selected from those with at least 90% women and at most 10% women workers, respectively.Source: D Evaluation Datasets and Data Splits · p. 19

D.1 Data Distributions and Splits

The appendix defines task-specific in- and out-of-distribution data splits, then describes how candidate responses are generated, scored, ranked, and used to fit a weighted reward model.

D.1 Data Distributions and Splits

  • Political, sycophancy, discrimination, and helpfulness/harmfulness evaluations use task-specific in- and out-of-distribution prompts, with held-out prompts reserved to assess generalization.The in-distribution prompts are divided into reward-fitting, fine-tuning, and evaluation subsets; out-of-distribution prompts are used exclusively for evaluation.Source: §D.1 · pp. 19–20

E Candidate Generation, Rubric Scoring, and Preference Elicitation

  • Candidate pools vary along target behaviors; rubric scores form 89-dimensional embeddings, and constitution-guided rankings supply preference data for fitting the reward model.Most rubric items use exemplar-based grading, while three use counterfactual probes.Source: E Candidate Generation, Rubric… · pp. 20–21

F Fitting the Reward Model

  • A linear Plackett–Luce model assigns reward as a weighted sum of rubric scores and estimates the weights by maximizing the likelihood of observed rankings.The fitted coefficients describe how the model trades off rubric dimensions when predicting elicited preferences.Source: F Fitting the Reward Model · p. 21

G Comparison with Black-Box Reward Models

The comparison evaluates rankings from the rubric-based reward model against open-weight reward models. Agreement is comparable for Gemma and Qwen, lower for GPT-OSS, and should not be treated as evidence of better alignment.

G Comparison with Black-Box Reward Models

  • For helpfulness and harmfulness, rubric-based rankings agree with neural models about as closely as neural models agree with one another for Gemma and Qwen; GPT-OSS agreement is lower.GPT-OSS correlations are 0.35–0.45.Source: G Comparison with Black-Box Reward… · p. 21, Figure A2 · p. 22
  • Sign-constrained rewards retain near-comparable agreement with unconstrained rewards: 0.69–0.91 for Gemma and Qwen and 0.37–0.50 for GPT-OSS.These ranges are for the helpfulness and harmfulness evaluation.Source: G Comparison with Black-Box Reward… · p. 22
  • Closer agreement with existing reward models is not necessarily desirable, so ranking correlations are reassurance about plausibility rather than the primary alignment assessment.The primary assessment concerns behavior of models trained against the rewards.Source: G Comparison with Black-Box Reward… · p. 22

H Training Procedures

The section outlines two fine-tuning settings: training language-model policies using reward-selected responses, and training black-box reward models on biased preference data.

H Training Procedures

  • The procedures cover fine-tuning language-model policies on responses selected by a reward model and evaluating their behavior.Source: H Training Procedures · p. 22
  • A second procedure fine-tunes black-box reward models on biased preference data to examine whether learned bias passes into policies trained from their selections.Source: H Training Procedures · p. 22
  • The section first describes policy training and evaluation, then the reward-model fine-tuning used in label-bias experiments.Source: H Training Procedures · p. 22

H.1 Policy Fine-Tuning

For each base model, the fitted rubric reward selects response targets from candidate pools under modified coefficient settings, after which each selected dataset is fine-tuned separately. Steering coefficients alter reward dimensions, while supplementary data and training and decoding settings are held fixed across settings.

H.1 Policy Fine-Tuning

  • With an infinite coefficient, selection first optimizes the targeted criterion and then uses the remaining reward to choose among tied candidates.Infinite political lean applies only to political prompts; settings with both coefficients infinite are omitted.Source: §H.1 · p. 23
  • Each coefficient setting selects responses using a modified reward, then fine-tunes a separate policy from the corresponding base model.Rubric scores are computed once and reused across coefficient settings.Source: §H.1 · pp. 22–23
  • Positive βlean favors conservative over liberal argumentation, while positive βhelp rewards four helpfulness items without changing any harm-item coefficient.Steering coefficients are normalized by the sample standard deviation of the unmodified reward on the corresponding fine-tuning candidates.Source: §H.1 · p. 23
  • Positive βsycophancy penalizes both sycophancy and performative independence in the modified reward.The sycophancy coefficient is normalized by the standard deviation of the unmodified reward on feedback prompts.Source: §H.1 · p. 23
  • The selected responses are supplemented with fixed Tülu-3 SFT examples, comprising 20% of political/helpfulness and sycophancy training rows and about 9.5% of discrimination rows.The same supplementary examples are used for every coefficient setting and base model.Source: §H.1 · p. 23
  • Policies are fine-tuned with supervised learning, and evaluation uses fresh self-generated responses with fixed decoding settings across coefficient settings.Evaluation includes 200 MMLU questions and 200 strict IFEval prompts; rescoring IFEval can change strict scores by about two percentage points because its checker is not fully deterministic.Source: §H.1 · pp. 23–24

H.2 Black-Box Reward Model Fine-Tuning

This section documents the prompts, grading procedures, rubrics, and question-set construction used for black-box reward-model fine-tuning and evaluation. It covers controlled gender-bias injection, harmfulness, task helpfulness and political-reception criteria, and the generation and validation of a balanced set of political questions.

H.2 Black-Box Reward Model Fine-Tuning

  • At q = 0–1, gender-stereotype preference labels were injected with nested flips, and rubric rewards used 200 pairs while neural rewards used all 400.One uniform draw per pair was shared across doses; stereotype-aligned preferences rose from 0.52–0.58 at q = 0 to 1 at q = 1.Source: §H.2 · pp. 24–25

I.2 Constitution-guided preference elicitation

  • The prompt set includes constitution-guided selection among candidate responses and separate templates for harmful-request refusals, seeded ratings, and political engagement.The political prompt asks for a brief argument for one side and a strong, well-developed argument for the other.Source: §I.1.1 · p. 26, §I.2 · p. 27, §I.1.3 · p. 27

I.3 Exemplar-based rubric grading

  • Rubric-grading prompts require a level for every item, use scale anchors for absent behaviors, and ask graders to ground judgments in response evidence.Quality scores increase with the good trait, while violation scores increase with the bad trait.Source: §I.3 · p. 28

I.4.1 Compose (strip the user’s stance / demographic cue)

  • Counterfactual prompts remove or neutralize user-stance and demographic pressure while preserving task-relevant facts, then compare response substance with a fresh reply.The grading instructions distinguish movement toward or away from the user's wishes from magnitude of change, and exclude presentational differences.Source: §I.4.1 · p. 29, §I.4.2 · p. 30

I.5.1 Harm (1-5)

  • The harm rubric scores from 1 to 5 according to how much a response materially advances a harmful objective, distinguishing partial harmful compliance from full fulfillment.Disclaimers do not reduce a score when harmful content is present, while refusals or safe redirections should not be scored as harmful merely for discussing the topic.Source: §I.5.1 · pp. 31–32

I.5.2 Helpfulness (0-3)

  • The helpfulness rubric scores from 0 to 3 based only on whether the response gets the requester’s job done, excluding safety, legality, morality, and whether help was appropriate.It ranges from no help to directly actionable full help; warnings or caveats do not change the score when the requested substance is delivered.Source: §I.5.2 · p. 32

I.5.3 Political lean

  • The political-lean rubric labels a response liberal, conservative, neither, or unclear based on its substantive position and likely relative reception, not factual correctness or fairness.Lean strength ranges from 0 for neither to 3 for a strong liberal or conservative lean, and is null when the lean is unclear.Source: §I.5.3 · pp. 33–34

I.6 Movie-choice prompt

  • The movie-choice prompt asks respondents to choose between two titles based on what they expect a named friend would like, and requires a one-line A-or-B answer.The specified response format is “Choice: A” or “Choice: B.”Source: §I.6 · p. 34

I.7 Political-question generation

  • The political-question bank was designed to contain 2,000 concise natural-language questions with left, right, or neutral framing, approximately balanced across orientations and a broad range of issue families.Its generation prompt asked for four CSV columns and required at least 12 issue families.Source: §I.7 · pp. 34–36
  • Political prompts should address contested evaluative issues, avoid excluded or extreme content, and be classified by their actual wording rather than topic alone.The guidance also says not to name partisan candidates or politicians unless necessary.Source: §I.7 · p. 37
  • Prompts should use natural, varied, symmetric ideological language without systematically favoring one side or mechanically templating every topic.Topics may recur with different framings, but the dataset should feel diverse rather than templated.Source: §I.7 · p. 38
  • The final output must be a CSV with exactly 2,000 data rows, four specified columns, balanced lean categories, unique prompts, and valid field and prompt formats.Each topic must be at most three words, and every prompt must end with a question mark.Source: §I.7 · p. 38

J Additional Figures and Tables

The additional figures examine reward-model agreement, targeted steering, and the robustness of policy behavior under sign constraints and injected preference bias. They also show that steering leaves MMLU relatively stable, while Strict IFEval differences vary across base models.

J Additional Figures and Tables

  • The rubric-based reward rankings generally agree less with neural reward models than Skywork and QRM agree with each other on the listed evaluations.Under sign constraints, agreement on harmfulness–helpfulness is nearly the same as under the unconstrained reward.Source: Figure A4 · p. 40, Figure A7 · p. 42, Figure A8 · p. 43
  • Varying helpfulness has little effect on political lean, and varying political lean has little effect on safety, with some conservative models becoming more liberal as helpfulness rises.The figure notes that observations with infinite coefficients for both dimensions are ill-defined and omitted.Source: Figure A5 · p. 41
  • Bias in gender preferences passes into policies trained with neural rewards and transfers to held-out movie recommendations, while rubric-reward policies remain comparatively stable.The figure plots gender-aligned recommendations against the probability of flipping preferences toward stereotypical recommendations.Source: Figure A6 · p. 42
  • Political lean shifts with its steering weight, while helpfulness and safety trace a tradeoff as the helpfulness weight varies.The sycophancy framing gap decreases as its penalty increases, though some sensitivity to user opinion remains.Source: Figure A9 · p. 44, Figure A10 · p. 44
  • Across coefficient settings, MMLU scores vary relatively little; increasing the sycophancy penalty produces no consistent decline.This pattern is shown for policies after political-lean and helpfulness steering and sycophancy penalization.Source: Figure A12 · p. 46
  • Strict IFEval differences from the untrained reference vary by base model: Gemma generally scores above it, while Qwen and GPT-OSS generally score below.The same model-dependent direction appears under the sign-constrained reward, without a consistent decline as the sycophancy penalty rises.Source: Figure A13 · p. 47, Figure A15 · p. 49
  • Under the sign-constrained reward, MMLU varies little across coefficient settings, though differences from the reference vary across models and experiments.The related steering panels show relatively little MMLU variation and no consistent decline as sycophancy penalties increase.Source: Figure A12 · p. 46, Figure A14 · p. 48
  • Injected gender bias yields comparatively little variation in MMLU and Strict IFEval across base models and reward-model settings.For Strict IFEval, Gemma generally scores above the reference, Qwen remains near it, and GPT-OSS is generally near or below it.Source: Figure A16 · p. 50, Figure A17 · p. 51
Loading 2609.33086v1…