PaperlayerView exact revision on arXiv

Source-linked AI summary

What Is Lost in Post-Training? Default Collapse and the Loss of In-Context Steerability Across Diverse Perspectives

Jessica Dierking, Itai Shapira, Niclas Boehmer

arXiv:2610.02614v1cs.LG

TL;DR

Prior work has focused on post-training’s effect on default views, leaving less evidence about whether models can adapt to in-context perspectives. The paper tests one-sided training pressure in controlled experiments and proposes a reward objective constrained by a target perspective distribution. It finds that pressure both strengthens the trained default and weakens alternative-perspective steerability, pointing to a tension between single-view alignment and serving diverse users.

  • Problem

    Prior evidence on post-training’s narrowing largely evaluates neutral-prompt defaults, which cannot establish whether models can still enact alternative perspectives when explicitly steered.

    Source: §1 · p. 1

  • Method

    The paper trains models toward one pole of a single perspective dimension under controlled conditions and proposes a reward-maximizing objective constrained to a prescribed perspective distribution.

    Source: §1 · pp. 2–3

  • Results

    Increasing one-sided pressure across SFT, RLHF, and DPO strengthens the trained default and reduces steerability, largely alongside failures to recognize the requested perspective’s matching response.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings point to a structural tension between aligning models to a single set of values and preserving the technical capacity to serve heterogeneous populations.

    Source: §5 · p. 9

    The recognition-only fine-tuning uses no steering prompts, but similar wording between recognition and steering questions means improvement may partly reflect transfer between the tasks.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

AI models serving a heterogeneous population must act on the principles appropriate to each user and context. While post-training has been shown to narrow the views large language models express, prior work has focused on default behavior rather than the ability to adapt to in-context information. We show that post-training also degrades a model's ability to be steered in-context toward perspectives it was not trained to favor. In controlled experiments, we fine-tune models toward one side of cultural-value disagreements and evaluate checkpoints throughout training. The trained side becomes increasingly dominant in ordinary use, while the ability to recognize and faithfully enact the opposing view declines. These findings point to a tension between prioritizing a single set of values and preserving the technical capacity needed to serve diverse stakeholders. Finally, we propose and analyze an alternative objective that maximizes reward subject to a prescribed distribution over expressed perspectives, and present stance-distribution matching as a practical implementation.

1 INTRODUCTION

Prior evidence on post-training’s narrowing largely concerns neutral-prompt defaults, leaving open whether models can still enact perspectives when explicitly steered. Controlled experiments show that increasing one-sided training pressure narrows defaults and weakens steerability, while the paper also proposes an objective for preserving a prescribed perspective distribution.

1 INTRODUCTION

  • Prior evaluations largely measure neutral-prompt defaults, which cannot distinguish an intended default from an inability to enact alternatives when context calls for them.Sensitivity to in-context information is needed for one model to serve heterogeneous users.Source: §1 · pp. 1–2
  • The controlled experiments train toward one pole of a single measurable perspective dimension while holding other factors fixed, isolating the effect of a one-sided signal rather than modeling deployed LLM training.The setup tracks default shifts and recognition and following of the opposing pole.Source: §1 · p. 2
  • Across SFT, RLHF, and DPO, increasing one-sided post-training pressure strengthens the trained perspective by default and progressively reduces steerability toward other perspectives.Both effects generalize to held-out domains and free-form answers, although training uses multiple-choice questions.Source: §1 · p. 2
  • Steerability loss is largely accompanied by failure to recognize the response matching a requested perspective, while recognition-only fine-tuning restores much of the lost steerability.That fine-tuning mostly preserves default behavior; refusal accounts for only a small share of steerability loss.Source: §1 · p. 2
  • The proposed pluralism target specifies how often perspectives should appear, and the paper maximizes KL-regularized RLHF reward subject to keeping the expressed distribution near that target.The target applies to prompts where multiple perspectives should be represented.Source: §1 · p. 3
  • The constrained policy theoretically reweights perspectives while preserving RLHF-optimal responses within each perspective; SDM uses an LLM judge and PPO-style updates and improves reward on one dimension while moving another stance distribution closer to target.The desired perspective distribution is specified independently of the reward objective.Source: §1 · p. 3

2 PRELIMINARIES

The preliminaries define post-training pressure and the policy notation, then describe SFT, KL-regularized reward optimization, RLHF, and DPO. Figure 1 illustrates how training progress is measured across default choices, explicit steering, and recognition.

2 PRELIMINARIES

  • Post-training pressure is the strength of the push toward a target behavior, increased by more training steps or weaker regularization toward the base policy.Source: §2 · p. 3
  • The notation treats prompts and responses as spaces, models a policy as a response distribution given a prompt, and uses multiple-choice prompts with legal answer options for training.Source: §2 · p. 3
  • Hard-label SFT starts from the base policy and minimizes cross-entropy on a target response, assigning that response probability one at the unconstrained optimum on the training support.Source: §2 · p. 3
  • KL-regularized reward optimization maximizes expected reward while penalizing deviation from the base policy; increasing inverse temperature β weakens that penalty and amplifies reward gaps.The prompt-wise optimum reweights the base policy toward higher-reward responses.Source: §2 · p. 3
  • RLHF learns a reward model from human preferences and optimizes it, while DPO fits the policy directly to preference data for the same objective.On multiple-choice prompts, the legal-option distribution permits exact finite-sum evaluation and differentiation rather than response sampling.Source: §2 · p. 3
  • Figure 1 tracks the trained-pole choice on neutral prompts, following each pole under explicit steering, and correctly identifying matching responses as training progresses.Curves average 12 runs per model across six cultural-value dimensions and both poles.Source: Figure 1 · p. 4

3 DEFAULT COLLAPSE AND STEERABILITY LOSS

The experiments show that one-sided post-training shifts models toward the trained perspective by default and weakens their ability to follow an opposing perspective in context. Recognition failures largely accompany steering failures, and recognition-focused fine-tuning restores much of the lost steerability while mostly preserving the trained default.

3.1 EXPERIMENTAL SETUP

  • Using CDEval’s six bipolar cultural dimensions, the authors train toward one pole and track default choices and instructed responses on held-out multiple-choice and free-form questions.The experiments include six dimensions, two training poles per dimension, and evaluation on questions from held-out domains and topics.Source: §3.1 · pp. 4–5

3.2 EFFECTS OF ONE-SIDED POST-TRAINING

  • Increasing one-sided post-training shifts default behavior toward the trained pole and progressively weakens opposing-pole steering, including on held-out domains and, with further training, in free-form responses.Default collapse also appears when the model initially favors the other pole, and across the evaluated training setups.Source: §3.2 · pp. 5–6

3.3 ANALYSIS OF STEERABILITY LOSS

  • Recognition and steering often decline together; most held-out questions with recognition failure also fail steering, while a smaller set recognizes the requested view but does not follow it.The recognition and steering measures often decline at similar rates across training checkpoints.Source: §3.3 · pp. 6–7, Figure 2 · p. 6
  • Recognition-only SFT raised recognition and steerability from near zero to over 90% on held-out questions after a few updates, while largely retaining the trained default.The additional training used recognition prompts rather than steering prompts, but similar wording means task transfer may partly explain the steerability gain.Source: §3.3 · p. 6

4 COUNTERING DEFAULT COLLAPSE

Because one-sided post-training can make alternative perspectives harder to elicit, the section proposes maximizing reward while keeping perspective frequencies near a prescribed target. It analyzes the constrained objective and introduces stance-distribution matching (SDM) as a practical implementation.

4 COUNTERING DEFAULT COLLAPSE

  • The proposed target specifies the rough mix of perspectives desired by default on prompts where pluralism is wanted, while retaining the standard RLHF objective as the optimization goal.The section frames this as an alternative to simply reducing post-training pressure, which could also weaken its desired effects.Source: §4 · p. 7

4.1 THEORETICAL ANALYSIS

  • The constrained optimum changes how often perspectives are expressed while preserving the RLHF-optimal response distribution within each perspective.When active, the perspective distribution is a normalized geometric interpolation between the RLHF-induced distribution and the target.Source: §4 · p. 7, §4.1 · p. 8
  • Among policies satisfying the target constraint, the solution is closest in KL divergence to the unconstrained RLHF optimum, which remains optimal if it already satisfies the constraint..Source: §4.1 · p. 7
  • Every perspective with positive target weight retains positive limiting probability as reward pressure increases, though reward gaps and η can leave it with very little probability.The guarantee holds when η remains constant; the limiting distribution need not equal the target.Source: §4.1 · p. 8

4.2 FROM THEORY TO TRAINING: STANCE-DISTRIBUTION MATCHING

  • SDM uses LLM judges to identify relevant prompts and rate responses, then applies PPO-style updates using a pooled stance distribution against the target.Pooling across relevant prompts replaces per-prompt estimates and imposes a population-level penalty.Source: §4.2 · p. 9
  • Across two settings, SDM concentrated the default on the rewarded pole while moving the stance distribution on another dimension closer to its target.This was a preliminary evaluation using Llama3-8B-SFT as the base model.Source: §4.2 · p. 9

5 DISCUSSION

The discussion frames steerability as a technical capacity needed alongside value selection, while noting that controlled one-perspective experiments leave more complex interactions for future study. It reviews related work and describes the CDEval benchmark and experimental appendices.

5 DISCUSSION

  • Applying post-training toward a single viewpoint negatively impacts steerability, creating tension between aligning to one set of values and serving heterogeneous populations.Steerability is necessary but not sufficient for alignment; recognition tuning and distributional objectives are discussed as ways to address related costs.Source: §5 · p. 9
  • The experiments test a directional hypothesis in a controlled one-perspective setting; real interactions may involve multiple values and perspectives that emerge during conversation.The authors identify broader values and conversational settings as future work, along with studying target selection and steerability under combined recognition tuning and distributional objectives.Source: §5 · p. 9

A.1 DEFAULT COLLAPSE AND IN-CONTEXT STEERABILITY

  • Prior work documents concentrated defaults and limited steerability, while this paper analyzes how post-training pressure relates to steerability loss and recognition.The appendix says earlier methods for diversity or user-specified distributions did not provide a controlled trade-off between reward maximization and constraining outputs to a specified target distribution.Source: §A.1 · p. 19, A EXTENDED RELATED WORK · p. 19

A.2 DISTRIBUTIONAL TARGETS AND REWARD OPTIMIZATION

  • Earlier distributional approaches use belief-annotated data, calibrate output statistics, or preserve or increase diversity, but do not directly solve the paper’s reward-and-target-distribution problem.The reviewed methods vary in their data requirements and objectives; some lack an externally specified reward or a user-specified target distribution.Source: §A.2 · pp. 19–20

B EXPERIMENTAL SETUP

  • The experimental appendix covers dataset and split construction, training settings, evaluation protocols and metrics, and datasets considered but not used.The section introduces extended related work on default collapse, steerability, and distributional targets.Source: A EXTENDED RELATED WORK · p. 19, B EXPERIMENTAL SETUP · p. 20

B.1 CDEVAL: DIMENSIONS AND SPLITS

  • CDEval uses opposing perspectives on everyday decisions across six cultural dimensions and seven domains, with work and family held out alongside a question-level split.The evaluation tests transfer within CDEval; the authors caution that its six dimensions are not exhaustive and do not support claims about cultural behavior outside the benchmark.Source: §B.1 · pp. 20–21

B.2 TRAINING METHODS AND SETTINGS

The appendix describes hard-label SFT, recognition tuning, RLHF, and DPO, with training pressure varied across configurations. Runs use controlled task ordering, though each configuration is a single run.

B.2 TRAINING METHODS AND SETTINGS

  • Each configuration is a single training run, with one optimizer update per task and a seeded task order shared across runs toward opposing poles.The appendix notes that this design does not estimate variability across seeds.Source: §B.2 · p. 22
  • Hard-label SFT trains separate models for each dimension and pole against a target distribution assigning probability 1 to the target response; recognition tuning continues from the final SFT model.Recognition tuning uses recognition probes rather than value tasks.Source: §B.2 · p. 24
  • RLHF uses learned reward models trained with preference data from annotator populations that disagree to varying degrees, then optimizes reward with a reference policy.The reported reward-pressure coefficient β indexes independent runs starting from the same base model.Source: §B.2 · p. 24
  • DPO fine-tunes from the base model without an SFT warm-up, preferring the target-pole response and varying reciprocal β so larger values indicate stronger preference pressure.Each setting trains for three epochs and retains an end-of-epoch checkpoint.Source: §B.2 · p. 24
  • The configurations include KL-regularized reward, token cross-entropy, DPO logistic loss, and two-option cross-entropy objectives, with SFT pressure tracked by progress and DPO/RLHF pressure by β.The tables list training-family settings and the checkpoints or pressure settings used in the plotted trajectories.Source: Table B.3 · p. 23, §B.2 · p. 25, Table B.4 · p. 25

B.3 EVALUATION PROTOCOLS AND METRICS

The evaluation probes checkpoints on training and held-out questions under default, steering, and recognition prompts. It reports multiple-choice and free-form scoring approaches.

B.3 EVALUATION PROTOCOLS AND METRICS

  • Every checkpoint and the base model are probed on training and held-out questions using default, steering, and recognition prompts in both display orders.The protocol uses ten prompts per question.Source: §B.3 · p. 25
  • Multiple-choice scores average first-token probabilities across answer orders, while free-form scores use a seven-point stance judge.The evaluation table assigns each construct a distinct condition and scoring rule.Source: Table B.5 · p. 26
  • The evaluation distinguishes default behavior, steering, and recognition rather than treating them as a single measure.Each construct uses its own condition and rule.Source: §B.3 · p. 25, Table B.5 · p. 26

B.3.1 MULTIPLE-CHOICE METRICS

The appendix defines multiple-choice and free-text metrics, discusses dataset selection, and identifies additional empirical results reported using both scoring formats.

B.3.1 MULTIPLE-CHOICE METRICS

  • Multiple-choice default collapse measures how often the trained pole scores higher under a neutral prompt, while steerability and recognition test whether each requested pole scores higher under its corresponding prompt.Scores average the two answer display orders; the appendix cautions that probability mass on the answer letters can shrink substantially in some trained setups.Source: §B.3.1 · p. 26

B.3.2 FREE-TEXT METRICS

  • Free-text answers are rated on a seven-point stance scale, with mirrored scores for low-pole training and pooled results across trained poles and steering instructions.Only valid answers enter the metrics, and the scale distinguishes following, ignoring, or arguing against the steering instruction.Source: §B.3.2 · pp. 26–27

B.4 DISCUSSION OF DATASETS

  • The dataset discussion requires recurring perspectives, distinct alternative answers, and valid positions; it explains why several preference, moral-judgment, and survey datasets do not meet those criteria.The appendix says the selected datasets satisfy the experimental requirements or add little to the datasets already used.Source: §B.4 · p. 27

C ADDITIONAL EMPIRICAL RESULTS

  • The additional-results appendix reports results using multiple-choice answer-letter scores and free-form seven-point stance scores.The metrics are defined in Appendix B.3.Source: C ADDITIONAL EMPIRICAL RESULTS · p. 27

C.1.1 HARD-LABEL SFT

Hard-label SFT strongly shifts the default toward the trained pole while sharply reducing recognition and steerability, including on free-text answers. These effects also transfer to held-out domains.

C.1.1 HARD-LABEL SFT

  • After three epochs, hard-label SFT reaches 100% default collapse, while recognition falls to 11% and steerability to 0.9%.The pattern holds across all twelve dimension–pole units, including units where training opposes the base model’s initial preference.Source: §C.1.1 · p. 27
  • Without any free-text training examples, SFT shifts the free-text default from 4.00 to 6.09, about 70% of the distance from balance to the scale’s end.The default reaches 5.54 after 48 updates, 5.90 after 251, and 6.11 after 1,314 updates.Source: §C.1.1 · p. 29
  • In free text, counter-pole compliance falls from 6.28 to 5.12, while trained-pole compliance remains near 6.53–6.59.Multiple-choice steerability falls from 88% to 0.9% across the same held-out checkpoints; for GPT-4.1 mini, counter-pole free-text steerability declines only at the last two checkpoints.Source: §C.1.1 · p. 29, Figure C.3 · p. 30

C.1.2 TRANSFER TO HELD-OUT DOMAINS

  • Final SFT models show 100%, 99%, and 99% default collapse on unseen questions, Work, and Family, with steerability at 0.5%, 1%, and 1%.Recognition on those same splits is 10%, 12%, and 11%, respectively.Source: §C.1.2 · p. 30
  • Across the three held-out splits, the collapse and losses in recognition and steerability are similar; the Work and Family domains were held out entirely from training.Figure C.5 likewise reports little difference between unseen questions from trained domains and the unseen Work and Family domains.Source: §C.1.2 · p. 30, Figure C.4 · p. 30, Figure C.5 · p. 31

C.1.3 RLHF AND DPO

Across RLHF and DPO, stronger post-training pressure is associated with increasingly trained-pole defaults and declining recognition and steerability. RLHF’s weak-pressure and free-text results qualify how those changes appear across settings.

C.1.3 RLHF AND DPO

  • With RLHF, default collapse rises from 52% at β = 5 to 92% at β = 1000, while recognition and steerability decline with training at every β and faster at larger β.At β ≥50, default collapse exceeds 90% except at β = 500, where it is 86%.Source: §C.1.3 · p. 31
  • At β ≤20, RLHF defaults can remain on the base model’s favored pole regardless of the trained pole; answer mass is also lowest at these weak-pressure rungs.At β = 10, answer-letter mass is 0.13 and option-sentence mass is 0.19.Source: §C.1.3 · pp. 31–32
  • In free-text RLHF, the final default rises from 4.42 at β = 5 to 6.16 at β = 1000, but steerability loss follows training length rather than β.Final steerability ranges from 4.30 to 5.50; valid free-text answer share is lowest at β = 500 and 1000, at 0.66 and 0.67.Source: §C.1.3 · pp. 31–32, Table C.1 · p. 32
  • At β = 1000, default collapse is 93% for 75/25 labels and 70% for 55/45 labels, versus 92% for unanimous labels.The 55/45 reward models are near chance with median reward gap |∆| ≤0.01, consistent with their weaker collapse.Source: §C.1.3 · p. 32, Table C.2 · p. 34
  • In DPO, default collapse is near complete at every β, while recognition and steerability decline faster and further at larger β.Recognition and steerability fall below 25% at β = 20 and remain above 80% at β = 0.50.Source: §C.1.3 · p. 32, Figure C.10 · p. 35

C.2 RECOGNITION TUNING AND RECOVERY

Training collapsed SFT models on recognition probes alone recovers recognition and steerability while largely preserving the trained-pole default. Recovery extends across units and held-out domains, though free-text answer validity varies.

C.2 RECOGNITION TUNING AND RECOVERY

  • Recognition tuning restores recognition to 94% and steerability to 95%, while default collapse remains high at 90%.The models start from collapsed SFT checkpoints and train only on recognition probes; seven of twelve final models place less than 0.20 probability on the two answer letters.Source: §C.2 · p. 36, Figure C.11 · p. 36
  • Steerability recovers to at least 77% in every unit, while ten of twelve units retain at least 92% trained-pole default collapse.The default weakens to 74% for MAS-high and 35% for LTO-low.Source: §C.2 · p. 36, Figure C.12 · p. 37
  • Across three held-out splits, recognition reaches 96%, 93%, and 92%, and steerability reaches 97%, 94%, and 93%.Default collapse remains 91%, 90%, and 89% on those same splits.Source: §C.2 · p. 36, Figure C.13 · p. 38
  • After recognition tuning, both free-text steering directions reach 6.63 against the base model’s 6.28, while default collapse is 5.44.The mean valid-response share is 87%, but it falls to 21% for MAS-low and 6% at that run’s update 251.Source: §C.2 · p. 36, Figure C.14 · p. 38

C.3 EXTENSION TO DEMOGRAPHIC-GROUP SURVEY QUESTIONS

The survey extension finds that one-sided SFT increases trained-group defaults and recognition tuning recovers steerability while preserving those defaults. The deferred proofs formalize perspective-frequency constraints, their optimal policies, and their behavior under reward pressure, with scope conditions on matching and paraphrase invariance.

C.3 EXTENSION TO DEMOGRAPHIC-GROUP SURVEY QUESTIONS

  • On political and religious survey questions, hard-label SFT raises default collapse to 93% and 89%, while steerability is 31% and 32%; recognition tuning raises it to 66% and 78% while defaults remain 90% and 89%.The base model’s political default already favors the liberal group on 82% of held-out questions, with recognition at 13% and steerability at 46%; religion starts at 52%/48%, with recognition at 14% and steerability at 42%.Source: §C.3 · p. 39

D.1 SETUP AND KL IDENTITIES

  • For a fixed perspective marginal, the optimal response policy retains the RLHF policy’s conditional distribution within each perspective, leaving only the marginal to optimize.The response families are defined by the map assigning responses to perspectives.Source: §D.1 · p. 41

D.2 DISTRIBUTIONAL CONSTRAINTS AND THEIR PENALTIES

  • For tolerances strictly between zero and the unconstrained mismatch, a penalty coefficient can produce the same unique optimizer as the constrained problem, with the constraint active.At zero tolerance, the marginal must match q; when tolerance is at least the unconstrained mismatch, the RLHF optimum remains optimal.Source: §D.2 · pp. 41–42

D.3 OPTIMAL PERSPECTIVE REDISTRIBUTION

  • Increasing the penalty coefficient weakly decreases divergence from q; the exact-matching formulas apply to zero tolerance, not the general positive-tolerance optimum.For a fixed marginal, the optimal response policy uses the RLHF conditionals.Source: §D.3 · pp. 42–43

D.4 BEHAVIOR UNDER INCREASING REWARD PRESSURE

  • With fixed q and η, every perspective with positive target weight retains positive limiting mass as reward pressure grows, while its conditional policy concentrates on its highest-reward responses.The limit need not equal q, and the limiting mass can be small for large reward gaps or large η.Source: §D.4 · pp. 43–44

D.5 RESPONSE-LEVEL MATCHING AND PARAPHRASE INVARIANCE

  • Redistributing probability within a response family leaves its perspective mass unchanged, but the stated invariance does not cover changes to total base mass, rewards, perspective assignments, or scoring-map errors.Perspective-level matching penalizes frequency mismatch while leaving the original objective to select conditionals; its interpretation assumes disjoint response families.Source: §D.5 · p. 44

D.6 TARGET SUPPORT AND THE SAMPLED IMPLEMENTATION

The appendix distinguishes the ideal per-prompt objective from the sampled, pooled SDM implementation. Its optimal-policy characterization does not automatically apply to that implementation.

D.6 TARGET SUPPORT AND THE SAMPLED IMPLEMENTATION

  • With zero weights in a raw target, smoothing gives an effective full-support target; restricting perspectives instead conditions the η → ∞ RLHF policy on the remaining support.That conditioned policy need not be the original RLHF policy.Source: §D.6 · p. 45
  • For a differentiable full-support policy and fixed σ_x, the appendix gives the exact gradient contribution of the auxiliary penalty.The expected score gradient makes the constant from differentiating p log p vanish.Source: §D.6 · p. 45
  • The implementation matches distributions pooled across relevant prompts, a relaxation of the per-prompt objective whose optimum is not characterized by the appendix’s analysis.A pooled match can allow individual prompts to favor different poles.Source: §D.6 · p. 45, E SDM: IMPLEMENTATION AND EXPERIMENTS · p. 45
  • Sampled histograms, separate malformed-response handling, PPO clipping, and group normalization mean updates are not guaranteed to attain the ideal optimum.Conditional preservation in Eq. 16 is not a proved training invariant or guarantee of in-context steerability.Source: §D.6 · p. 45

E.1 EXPERIMENTAL SETUP AND IMPLEMENTATION

The SDM experiments combine stance-distribution matching on one value dimension with reward maximization on another. The implementation estimates pooled stance distributions from judged samples and uses PPO-style updates.

E.1 EXPERIMENTAL SETUP AND IMPLEMENTATION

  • SDM trains on open-ended CDEval questions, placing a stance-distribution target on one dimension and a reward model on the other.Training prompts omit answer options and do not name either pole; targets use the judge’s 1–7 stance scale.Source: §E.1 · p. 45, Table E.1 · p. 45
  • All relevant prompts for a dimension share one target, so SDM matches the pooled distribution rather than each prompt’s distribution.By KL convexity, this relaxes the per-prompt objective; satisfying that objective implies the pooled constraint, but not vice versa.Source: §E.1 · p. 46
  • The pooled stance estimate uses an exponentially decayed histogram of judged samples, while malformed samples are excluded.The effective window is 256 samples.Source: §E.1 · p. 46
  • The target is smoothed separately to provide full support even when raw target weights are zero.The passage states that the results apply to the smoothed target.Source: §E.1 · p. 46
  • Well-formed responses receive REINFORCE credit for distribution matching; malformed responses receive a fixed credit below any usable response.The sampled credits and reference-policy regularizer estimate the gradient, using the running histogram and base policy; runs use η = 1 and β = 100.Source: §E.1 · p. 46
  • Questions rated at least 2 for relevance are stance-judged; GPT-5.6-Luna assigns scores from 1 to 7 or flags malformed responses.The experiments use this judge, though the authors note that better-validated scoring models could be substituted and remain a next step.Source: §E.1 · pp. 46–47
  • The runs use OpenRLHF’s Ray PPO trainer with clipped updates, group-normalized advantages, and a KL penalty to the frozen base policy.Each rollout batch supplies one optimizer step, yielding a critic-free, group-normalized policy gradient.Source: §E.1 · p. 47

E.2 RESULTS

This section reports how the two SDM runs and reward maximization affect judged-dimension stance distributions and defaults, and describes the experiment prompts, including narrowly defined opposing poles across six cultural-value dimensions.

E.2 RESULTS

  • Default probability on the rewarded pole rose from 43% to 97% on MAS and from 82% to 91% on IVR, while an untargeted dimension collapsed as before.The PDI and IDV penalties left reward maximization on the other dimension intact.Source: Figure E.1 · p. 47, §E.2 · p. 47
  • KL divergence fell from 0.58 to 0.24 on PDI and from 1.12 to 0.67 on IDV as judged-dimension stance distributions moved toward their targets.PDI shifted from a low-side concentration toward the target’s bimodal shape; on IDV, neutral answers fell from 24% to 14%, with smaller gains amid partial conflict with the IVR reward objective.Source: Figure E.2 · p. 48, §E.2 · p. 48, Figure E.3 · p. 49

F.1 TRAINING PROMPTS

  • Training prompts elicit the model’s usual perspective or test recognition of a stated principle, while open-ended prompts support reward-model scoring and SDM rollouts.The value-task prompt produces two tasks per question, one per display order; the recognition prompt produces four, one per pole and display order.Source: §F.1 · pp. 50–51

F.2 EVALUATION PROMPTS

  • Evaluation prompts distinguish default answers from steerability tests that ask the model to follow a specified decision principle in forced-choice or free-form responses.The forced-choice steerability instruction says the principle overrides other considerations; the free-form prompt asks the model not to reveal that it was briefed.Source: §F.2 · pp. 52–53

F.3 RELEVANCE AND STANCE-JUDGING PROMPTS

  • The SDM relevance judge routes questions based on whether value poles would produce different answers, and ratings of 2 or 3 receive distributional credit; a stance judge scores answer positions.The stance judge uses a seven-point scale between the two poles and assigns a validity category.Source: §F.3 · pp. 55–57
  • The hierarchy prompt contrasts accepting authority and status differences with equal standing and shared participation, excluding politeness or deference in tone.Its scope is rank and equality of standing and say, not fairness of outcomes, relationship closeness, or independence from a group.Source: §F.3 · p. 59
  • The individualism prompt contrasts autonomy and responsibility to oneself or immediate family with loyalty and obligations to the wider group.It assesses whose claim takes priority in conflict, not mere mentions of family or community, first-person wording, or freedom from constraint generally.Source: §F.3 · p. 59
  • The uncertainty-avoidance prompt contrasts preference for clarity and structure with comfort with ambiguity, excluding cautious or confident writing style.It concerns discomfort with unpredictability, not attachment to the past or orderliness for its own sake.Source: §F.3 · p. 60
  • The achievement prompt contrasts competition and material success with modesty, cooperation, care, and quality of life, not assertive or warm tone.Achievement means winning and getting ahead rather than effort, diligence, or competence.Source: §F.3 · p. 60
  • The time-orientation prompt contrasts long-term planning and future rewards with traditions, present obligations, and immediate benefits, rather than mere references to past or future.It assesses valued time horizon; perseverance means persistence toward a future end, not self-discipline generally.Source: §F.3 · p. 61
  • The indulgence prompt contrasts enjoyment, leisure, and ordinary desires with duty, self-control, and socially regulated restraint, not permissive tone toward the reader.Freedom and restraint refer narrowly to ordinary desires, not broader liberty or social arrangements.Source: §F.3 · p. 61

Related papers

Loading 2610.02614v1…