PaperlayerView exact revision on arXiv

Source-linked AI summary

How Does Post-Training Shape LLM Value Preferences?

Yang Liu, Shaoyang Xu, Masahiro Kaneko, Chengzhi Zhong, Wenxuan Zhang, Chenhui Chu

arXiv:2610.33371v1cs.CL

TL;DR

The paper asks how opinion mixtures and post-training paths jointly shape LLM value preferences, a question relevant to understanding preference amplification and removal. Controlled experiments find that amplification, removal, intervention stability, and distillation transfer vary with post-training path and preference history.

  • Problem

    How opinion mixtures and post-training methods jointly shape LLM value preferences, their removal, intervention stability, and transfer through distillation has not been systematically compared.

    Source: §1 · p. 1

  • Method

    The authors run controlled opinion-mixture experiments across post-training stages and evaluate preferences on covered and uncovered questions.

    Source: §2.4 · p. 3, §3 · p. 5

  • Results

    Post-training amplifies corpus imbalances on covered questions, while further preference removal depends on method and mixture, intervention stability varies with preference changes, and task-only distillation transfers preferences to students.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Opinion mixture and post-training method should both be considered when developing, evaluating, and reusing LLMs.

    Source: §9 · p. 10

    The claims concern how post-training on an opinion mixture changes model preferences, not the views of individual respondents or demographic groups as a whole.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

During post-training, large language models (LLMs) learn value preferences from corpora that may contain imbalanced proportions of opinions associated with different demographic groups. Amplifying these imbalances may further underrepresent some groups' opinions. However, how opinion mixtures shape value preferences across post-training paths remains unclear. To answer this question, we conduct controlled experiments using post-training corpora with different opinion mixtures and five post-training paths based on SFT, DPO, and GRPO. We find that post-training amplifies opinion imbalances differently across post-training paths. Once value preferences are amplified, whether further post-training can remove them depends on both the post-training method and the opinion mixture in the corpus. More concerningly, a model's stability under interventions depends on whether further post-training changes its value preference. Moreover, these value preferences are transferred to student models even when distillation uses corpora with no explicit opinion questions. These findings suggest that both the opinion mixture in post-training corpora and the choice of post-training method should be considered when developing, evaluating, and reusing LLMs. Our code and data are available at https://github.com/ku-nlp/llm-value-preferences.

1 INTRODUCTION

The paper studies how opinion mixtures and post-training stages jointly shape LLM value preferences, including whether learned preferences can be removed, withstand interventions, and transfer through distillation. Its findings show that post-training paths differ in amplification and removal, while preference stability and transfer also matter for model reuse.

1 INTRODUCTION

  • The study addresses gaps in systematic comparisons of how post-training method and opinion mixture jointly shape preferences, their removal, intervention stability, and distillation transfer.Prior work had shown that corpus composition and post-training methods can shape expressed values and preferences, but the effects of stage order and further training remained unclear.Source: §1 · p. 1
  • The authors use controlled OpinionQA corpora with varying group-opinion proportions and five paths based on SFT, DPO, and GRPO, evaluating covered and uncovered questions separately.The five paths are SFT alone or SFT followed by DPO, GRPO, or both in either order.Source: §1 · p. 2
  • All five post-training paths amplify corpus opinion imbalance on training-covered questions, while removal of an existing preference depends on both method and corpus mixture.Balanced-corpus SFT can restore equal alignment, but other methods do not guarantee it; DPO may require more opposing-group opinions than preferred-group opinions to remove a preference.Source: §1 · p. 2
  • Task-only distillation transfers teachers’ learned value preferences to students, creating a risk of unintended preference propagation.The authors also find that intervention stability depends on the relationship between first-stage and final-stage preferences; neutral or later-reversed preferences show higher flip rates.Source: §1 · p. 2
  • The authors argue that value preferences should be evaluated throughout LLM development and reuse, particularly for open-weight models undergoing multi-stage post-training and distillation.The paper notes these models may be further post-trained and distilled into smaller models of other families.Source: §1 · p. 2

2 EXPERIMENTAL SETUP

The experiments construct controlled opinion mixtures from demographic-group survey answers, apply varied post-training paths to base models, and compare preferences across covered and uncovered questions. Two scores characterize group alignment and whether that alignment amplifies the corpus imbalance.

2.1 OPINION POST-TRAINING CORPUS

  • OpinionQA supplies multiple-choice opinion questions; the corpus uses group majority answers, retaining questions where the two groups differ and augmenting selected questions with semantic variants.The retained sets contain 200 party, 156 education, and 104 religion questions; each selected question is expanded into 25 semantically equivalent variants for post-training.Source: §2.1 · pp. 2–3

2.2 MIXTURE PROPORTION

  • The mixture proportion p is the share of corpus examples reflecting group g1’s opinion, varied from 0.0 to 1.0 in 0.1 increments.The remaining examples reflect group g2’s opinion.Source: §2.2 · p. 3

2.3 MODELS

  • The main experiments post-train three base models—Llama-3.1-8B, Qwen3-8B-Base, and OLMo-2-7B—without prior instruction tuning.Using base models allows study of the controlled corpora without prior instruction tuning.Source: §2.3 · p. 3

2.4 POST-TRAINING

  • Each post-training stage starts from the preceding checkpoint, so the study compares SFT alone and four SFT-first sequences combining DPO and GRPO in different orders.SFT comes first because it teaches the model to provide an option number required by later stages and evaluation.Source: §2.4 · p. 3

2.5 EVALUATION QUESTION SETS

  • Evaluation separates training-covered questions, whose semantics appear in the augmentation set, from training-uncovered questions not used for augmentation.This comparison examines preference changes under different levels of corpus coverage.Source: §2.5 · pp. 3–4

2.6 METRICS

  • The agreement score ranges from 0 to 1, with 0.5 indicating equal alignment; amplification measures whether agreement moves farther from 0.5 than the corpus mixture does.A positive amplification score indicates amplification, while a score at or below zero does not imply the absence of a group preference.Source: §2.6 · p. 4

3 VALUE AMPLIFICATION IN POST-TRAINING

Across controlled opinion mixtures, post-training amplifies imbalance on training-covered questions, while reasoning mitigates but does not eliminate amplification. The size of amplification varies by post-training path, and preference shifts also appear on uncovered questions.

3 VALUE AMPLIFICATION IN POST-TRAINING

  • All evaluated LLMs and demographic attributes show amplification on training-covered questions; on uncovered questions, preferences shift toward the corpus mixture most visibly from 0.3 to 0.7, then saturate toward the range’s ends.The uncovered-set shift is smaller than the covered-set amplification.Source: §3 · p. 5
  • Amplification persists on post-training examples themselves, and results remain when agreement uses full answer distributions rather than only highest-probability answers.These checks indicate the pattern is not caused solely by generalization beyond training examples or by reducing outputs to their highest-probability answers.Source: §3 · p. 5
  • Reasoning mitigates amplification, with agreement tracking corpus mixture approximately linearly, but preferences still increase with the prevalence of a group’s opinions.The same pattern holds on uncovered questions and the post-training corpus.Source: §3 · p. 5
  • Paths ending in GRPO show weaker amplification than SFT or paths ending in DPO; SDG is weakest, while the different outcomes for SDG and SGD indicate that stage order matters.The proposed explanation is that DPO can raise answer confidence and reduce sample diversity available to a subsequent GRPO stage.Source: §3 · p. 5

4 REMOVING EXISTING VALUE PREFERENCES THROUGH FURTHER POST-TRAINING

Whether further post-training removes an existing value preference depends on the method, the starting preference, and the final-stage corpus mixture. SFT and S2D offer more predictable control, but also show strong value amplification.

4 REMOVING EXISTING VALUE PREFERENCES THROUGH FURTHER POST-TRAINING

  • The experiment varies the final-stage corpus mixture across eleven proportions for each of eleven SFT starting checkpoints, evaluating all runs on the training-covered set.The starting checkpoints’ Democratic agreement scores range from 0.00 to 0.99.Source: §4 · p. 6
  • A second SFT removes initial preferences across all eleven starting checkpoints when the final-stage mixture is between 0.4 and 0.6.The agreement score crosses 0.5 in that range.Source: §4 · p. 6
  • DPO retains more of the initial preference; returning an extreme checkpoint to equal alignment requires a clear majority of opposing-group opinions.The stronger the initial preference, the larger the opposing-group proportion needed.Source: §4 · p. 6
  • Some paths including GRPO cannot restore equal alignment for some starting checkpoints under any final-stage mixture; SDG and SGD are less effective than S2D and S2G.The influence of the starting checkpoint is strongest for paths combining DPO and GRPO, particularly SDG.Source: §4 · p. 6
  • SFT and S2D provide the most predictable control over value preferences, but both also exhibit some of the strongest value amplification.For other paths, restoring equal alignment depends on the initial preference, corpus mixture, and sequence of methods.Source: §4 · p. 6

5 STABILITY OF VALUE PREFERENCES UNDER INTERVENTIONS

Intervention stability depends on a model’s post-training history, not just its final preference. Models with the same final-stage preference can have different flip rates depending on their first-stage preference and whether later stages changed it.

5 STABILITY OF VALUE PREFERENCES UNDER INTERVENTIONS

  • Table 1 reports mean flip rates by first-stage and final-stage preference groups, separately for two- and three-stage paths.Its blue cells denote inference-time interventions, red cells denote weight-changing interventions, and darker shading means higher flip rate.Source: Table 1 · p. 7
  • The flip rate evaluates how often an intervention shifts answers from the majority answers of the model-aligned group to those of the opposing group.The five interventions include three inference-time methods and two weight-changing methods.Source: §5 · p. 7
  • For inference-time interventions, checkpoints with a clear first-stage preference generally have lower flip rates than neutral checkpoints, even when final-stage preferences match.The methods used in later stages and the number of stages explain little variation in flip rate.Source: §5 · p. 7
  • For weight-changing interventions, flip rates are higher when later stages change the first-stage preference than when they preserve it.For TASKVEC, flip rates are highest when the final-stage preference reverses the first-stage preference.Source: §5 · pp. 7–8

6 TRANSFER OF VALUE PREFERENCES THROUGH TASK-ONLY DISTILLATION

Task-only distillation transfers teachers’ value preferences to student models, even on questions absent from the post-training corpus. Transfer occurs across different task datasets and answer formats, with its strength varying across teacher paths and student models.

6 TRANSFER OF VALUE PREFERENCES THROUGH TASK-ONLY DISTILLATION

  • The study distills teacher answers on 5,000 sampled questions from each of WinoGrande, MMLU, and GSM8K into three student models.Teachers are selected to align more strongly with one or the other demographic group, with comparable agreement scores within each attribute.Source: §6 · p. 8
  • Positive preference gaps appear across every teacher path, task dataset, demographic attribute, and student model, on both training-covered and training-uncovered sets.S2G produces the largest gap across teacher paths and S2D the smallest; transfer is strongest from religion-corpus teachers and weakest to Qwen3-1.7B-Base.Source: §6 · p. 8
  • Preference transfer also occurs on MMLU with alphabetic labels and GSM8K with free-form responses, not only on WinoGrande with numeric labels.The training-uncovered results show that transferred preferences extend to questions absent from the post-training corpus.Source: §6 · pp. 8–9
  • Task data labeled by LLMs should not be assumed value-neutral, and both teacher and student models should be evaluated for value preferences.This conclusion concerns task-only distillation where the corpus contains no explicit opinion questions.Source: §6 · p. 9

7 ROBUSTNESS AND SCOPE

Value amplification persists across several experimental variations, while general task performance and social-bias measures do not consistently track value alignment. The reported changes vary by setting and do not follow a consistent direction across attributes or benchmarks.

7 ROBUSTNESS AND SCOPE

  • Value amplification remains evident across random seeds, prompt templates, added general task data, full-parameter fine-tuning, and instruction-tuned starting models.The authors report this as evidence that amplification is not an artifact of a particular experimental choice.Source: §7 · p. 9
  • General task performance does not systematically vary with opinion mixture, and social-bias measures do not consistently track value-alignment changes.Some results vary with nearly one-group corpora or reasoning, without a consistent direction across attributes or benchmarks.Source: §7 · p. 9
  • Value preferences should not be interpreted as changes in general capability or as equivalent to behavior measured by established social-bias benchmarks.Post-training itself can change task performance relative to the base model.Source: §7 · p. 9

8 RELATED WORK

Prior work links corpora to LLM behavior and studies bias amplification, but has not systematically tested how opinion proportions and post-training paths jointly shape value preferences.

8 RELATED WORK

9 CONCLUSION

The experiments show that post-training can amplify opinion imbalances, that removing existing preferences depends on method and corpus mixture, and that preferences can respond differently to interventions and transfer through distillation. The findings point to considering both corpus opinion mixture and post-training method when developing, evaluating, and reusing LLMs.

9 CONCLUSION

  • Post-training amplifies corpus opinion imbalances, while preference removal depends on the method and mixture; learned preferences also vary in intervention stability and transfer through distillation.The authors recommend considering both opinion mixture and post-training method when developing, evaluating, and reusing LLMs.Source: §9 · p. 10

ETHICS STATEMENT

  • Group labels denote statistical categories, and the measured preferences concern model responses to an opinion mixture, not the views of individual respondents or demographic groups as a whole.A group’s majority answer is only its most common response to a question and does not represent every member.Source: ETHICS STATEMENT · p. 10
  • The findings have dual-use implications: opinion imbalances can be amplified, existing preferences can be difficult to remove on some paths, and task-only distillation can transfer preferences to students.The authors report these findings because developers and auditors need methods to detect and mitigate such preferences.Source: ETHICS STATEMENT · p. 10
  • Released checkpoints contain value preferences by design and are research artifacts for reproducing measurements, not models intended for deployment.The survey data come from the public OpinionQA release; no new human-subject data were collected or individual-level records redistributed.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors will release the post-training corpora, training and evaluation code, and stored model outputs used to compute reported numbers.Two additional random seeds are used to quantify run-to-run variation across all five post-training paths.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 OUTPUTS OF THE POST-TRAINED LLMS

  • On training-uncovered questions, neither-group answers average 7.7%–18.9%, versus at most 2.5% on covered questions except Qwen3-8B-Base after education DPO at 8.5%.The uncovered-set mean is always more than five percentage points higher than the covered-set mean.Source: §A.1 · p. 17
  • Normalization changes some score values, especially on uncovered questions, but amplification remains positive on covered and negative on uncovered questions under both calculations.The appendix concludes that excluding answers matching neither group does not change its amplification conclusions.Source: §A.1 · p. 17

B.1.1 VALUE EVALUATION DATASET

  • The evaluation selects questions whose groups’ majority answers lie on opposite sides of the scale midpoint, without requiring substantially different full answer distributions.The authors note that majority-answer evaluation can make groups appear more different; distribution-based scoring preserves overall conclusions but may change post-training-path rankings.Source: §B.1.1 · p. 20

B.2 DATA AUGMENTATION

Across post-training paths, models amplify corpus imbalances on training-covered questions, while reasoning weakens mixture sensitivity on training-uncovered questions. The section also details how the opinion corpora and training paths are constructed.

B.2 DATA AUGMENTATION

  • Including fixed answer options in the paraphrase prompt helps preserve the intended question and keeps its original options valid.Without the options, GPT-5.6-Sol misinterprets an incomplete question stem; with them, all five rewrites remain compatible with the options.Source: §B.2 · pp. 22–23, Figure 6 · p. 23

B.3 DATA ASSEMBLY

  • The corpus keeps prompts and example counts fixed across 11 mixture proportions, changing only which group’s majority answer supplies the training signal.The corpus contains 2,500 party, 1,950 education, and 1,300 religion examples.Source: §B.3 · pp. 23–24

B.4 REASONING CHAINS

  • Generated reasoning chains show no systematic group differences in length or hedging-expression use across the three demographic attributes.The chains support specified answers in SFT and DPO; in GRPO, the model generates its own reasoning before answering.Source: §B.4 · p. 24, §B.3 · p. 24

B.5 TRAINING OBJECTIVES

  • SFT trains on prompt and answer tokens, DPO uses the groups’ majority answers as preference pairs, and GRPO scores eight completions against one sampled target.For GRPO, a correct option receives a text-similarity-adjusted reward between 0.7 and 1; a wrong option receives no reward.Source: §B.5 · p. 25

B.6 STARTING CHECKPOINTS FOR THE FINAL POST-TRAINING STAGE

  • Final-stage experiments generally begin from checkpoints without a strong preference for either group, because preceding stages use balanced corpora.Only two starting-checkpoint agreement scores differ from 0.5 by more than 0.2.Source: §B.6 · p. 26, Table 7 · p. 26

C.1 FURTHER ANALYSES OF AMPLIFICATION

  • On training-covered questions, all five paths amplify corpus opinion imbalances; on training-uncovered questions, reasoning weakens mixture sensitivity and does not amplify the imbalance.On the post-training corpus, direct-answer evaluation shows similar trends to covered-set evaluation; neither shows amplification under reasoning.Source: §C.1 · pp. 26–27, Figure 9 · p. 27

C.2 OTHER ATTRIBUTES, BACKBONES, AND THE TRAINING-UNCOVERED SET

  • Amplification scores are consistently higher on training-covered than training-uncovered questions across the tested attributes, backbones, and paths.The comparison covers three demographic attributes, three backbones, and five post-training paths.Source: §C.2 · p. 27, Table 8 · p. 27

C.3 ANSWER CONFIDENCE

Answer confidence, distribution-based agreement, and preference removal vary across post-training paths. The reported intervention procedures and checkpoint groupings provide context for these comparisons.

C.3 ANSWER CONFIDENCE

  • On training-covered questions, confidence falls near a balanced corpus mixture by different amounts across paths; reasoning maintains high confidence across mixtures and paths.SDG confidence remains above 0.85 across the full mixture range, while SFT falls below 0.25 at balance.Source: §C.3 · p. 28, Figure 10 · p. 28

C.4 DISTRIBUTION-BASED AGREEMENT AND AMPLIFICATION SCORES

  • Amplification on training-covered questions persists with full answer distributions, but distribution-based amplification scores are lower than majority-answer scores for all five paths.The largest decrease occurs under SFT, where lower confidence near the balanced mixture can make highest-probability-only agreement overstate alignment.Source: §C.4 · pp. 29–30, Figure 11 · p. 29

D HOW INITIAL PREFERENCES AFFECT PREFERENCE REMOVAL

  • A second SFT removes preferences across tested starting checkpoints, while paths combining DPO and GRPO generally make equal alignment harder to reach, with effects varying across LLMs.Under SFT, agreement crosses 0.5 for all tested Qwen3-8B-Base and OLMo-2-7B starting checkpoints when the final corpus mixture is between 0.4 and 0.6.Source: D HOW INITIAL PREFERENCES AFFECT… · p. 30, Figure 12 · p. 30, Figure 13 · p. 31

E.1 THE FIVE INTERVENTIONS

  • The interventions include adding an opposing-group answer to prompts, directing the model to output that answer, adding a contrastive activation vector, scaling a final-stage update, or continued fine-tuning.The continued-fine-tuning intervention uses 40 examples from questions outside the evaluated set.Source: §E.1 · pp. 31–32

E.2 CHECKPOINTS IN EACH GROUP

  • Checkpoint notation records the sequence and mixture proportions of post-training stages; final preference groups are assigned using agreement scores rather than the final-stage mixture.The experiments include 297 Llama-3.1-8B checkpoints for the party attribute.Source: §E.2 · p. 32

E.3 FLIP RATES ACROSS POST-TRAINING PATHS

Flip-rate patterns vary with intervention, post-training history, and backbone. The final subsections describe task-only distillation and its evaluation setup.

E.3 FLIP RATES ACROSS POST-TRAINING PATHS

  • Most interventions have higher flip rates on training-uncovered than training-covered questions, whereas TASKVEC has similar rates across the sets.TASKVEC reduces the final-stage parameter update rather than explicitly pushing toward opposing-group answers.Source: §E.3 · p. 32, Table 11 · p. 34

E.4 FLIP RATES ACROSS BACKBONES

  • TASKVEC and CFT have higher flip rates when later stages change the first-stage preference than when they preserve it; ENDORSEMENT and OVERRIDE flip more often for neutral-start checkpoints.CAA’s relationship with preference history differs across backbones.Source: §E.4 · p. 33, Table 13 · p. 35
  • Mean flip rates under ENDORSEMENT and OVERRIDE vary more across backbones than those under TASKVEC and CFT; Qwen3-8B-Base is higher for the former pair.The comparison averages over 297 checkpoints from each backbone.Source: §E.4 · p. 33

F.1 DISTILLATION AND EVALUATION DETAILS

  • Students are trained to reproduce teacher responses on task-only corpora containing no opinion questions or explicit demographic references, then evaluated for task accuracy and preference transfer.The study uses three student models and evaluates preference gaps on each student’s teacher attribute.Source: §F.1 · pp. 34–35

F.2 TASK PERFORMANCE AND PREFERENCE TRANSFER

Distillation does not consistently improve downstream task accuracy, but teacher value preferences transfer to students trained on task-only corpora. Opinion post-training also changes answer-selection patterns more on WinoGrande than MMLU, without numeric labels being necessary for preference transfer.

F.2 TASK PERFORMANCE AND PREFERENCE TRANSFER

  • Distillation does not consistently improve task accuracy: MMLU accuracy rises for Llama-3.2-3B and OLMo-2-1B but falls for Qwen3-1.7B-Base.GSM8K accuracy rises for Llama-3.2-3B and falls for the other two students; WinoGrande remains around the 50% random-guessing baseline.Source: §F.2 · p. 36, Table 16 · p. 37
  • Teacher value preferences transfer to all three student models through task-only distillation, despite distillation corpora containing no opinion questions.Students trained on M+ responses align more strongly with group 1 than students trained on corresponding M− responses.Source: §F.2 · p. 36
  • Teacher option-selection frequencies shift more on WinoGrande than MMLU, with paired M+ and M− teachers differing by up to 65 percentage points on WinoGrande.For religion, M+ selects option “1” less often than M− across all five paths.Source: §F.2 · pp. 36–37, Figure 14 · p. 37
  • Preference transfer also occurs after MMLU and free-response GSM8K distillation, supporting that it is not driven by numeric option labels.Teacher option-selection frequencies change much less on MMLU than WinoGrande.Source: §F.2 · p. 37

G.1 ROBUSTNESS ACROSS RANDOM SEEDS

Robustness tests show that value amplification persists across random seeds, prompt templates, general-task co-training, fine-tuning parameterizations, and instruction-tuned starting models. Its measured strength varies with template, backbone, and starting preferences, while opinion mixture has no systematic relationship with general task performance.

G.1 ROBUSTNESS ACROSS RANDOM SEEDS

  • Across three seeds, the qualitative amplification pattern recurs; agreement-score RMS ranges from 0.012 to 0.076, and DPO has the lowest agreement-score RMS on all three backbones.GRPO has the largest RMS on Qwen3-8B-Base and OLMo-2-7B (0.064 for both), but not on Llama-3.1-8B.Source: §G.1 · p. 38, Table 17 · p. 39

G.2 ROBUSTNESS TO THE PROMPT TEMPLATE

  • Value amplification remains observable across prompt templates, but changing numeric labels or option order affects measured alignment more than changing question framing.Reversing option-content order reverses the direction of agreement-score change in most cases; option-order sensitivity also appears in the tested instruction-tuned models.Source: §G.2 · pp. 39–40, Figure 16 · p. 40

G.3 CO-TRAINING WITH A GENERAL CORPUS

  • Amplification persists when opinion examples are co-trained with up to five times as many MMLU examples, and also with ARC at a 1:1 ratio.Adding MMLU does not change which post-training paths amplify or reduce the corpus imbalance.Source: §G.3 · p. 41, Figure 17 · p. 41, Figure 18 · p. 41

G.4 FULL-PARAMETER FINE-TUNING

  • Full-parameter fine-tuning and LoRA both amplify preferences on imbalanced corpora, with stronger average amplification under full-parameter fine-tuning.The comparison averages agreement scores across the five post-training paths.Source: §G.4 · p. 42, Figure 19 · p. 42

G.5 STARTING FROM AN INSTRUCT MODEL

  • Amplification persists from instruction-tuned starting models, while DPO and GRPO curves differ more by starting point than SFT curves.The mean absolute difference between base-start and instruction-tuned-start SFT curves is 0.02 across mixtures, with a maximum difference of 0.07; initial preferences may explain some differences.Source: §G.5 · p. 42, Figure 20 · p. 42, Table 19 · p. 43

H.1 DOWNSTREAM TASK PERFORMANCE

  • Opinion mixture has no systematic relationship with general task performance, although post-training can change performance and most models score lower on MMLU and WinoGrande than their base models.Larger variations occur occasionally on MMLU, GSM8K, and IFEval, without a consistent pattern across models, paths, or benchmarks.Source: §H.1 · p. 43

H.2 SOCIAL-BIAS BENCHMARKS

Established social-bias benchmarks respond differently to post-training than the paper’s value-preference measures. Social-bias scores vary little across post-training paths overall, while mixture-related changes are concentrated in DiscrimEval rather than BBQ or CrowS-Pairs.

H.2 SOCIAL-BIAS BENCHMARKS

  • Social-bias scores are mostly similar across the five post-training paths, with standard deviations at most 0.014; reasoning generally lowers BBQ bias scores but raises DiscrimEval scores.CrowS-Pairs changes little between direct answering and reasoning.Source: §H.2 · p. 45, Table 21 · p. 45
  • BBQ and CrowS-Pairs scores remain relatively stable across opinion mixtures, while DiscrimEval varies particularly for party and group-dominated corpora.DiscrimEval explicit scores show the clearest path differences: SFT generally produces the largest changes, while paths ending in GRPO produce smaller changes.Source: Figure 21 · p. 45, §H.2 · p. 46
  • Value-preference measures and established social-bias benchmarks capture different aspects of model behavior and should not be treated as interchangeable.The reported social-bias scores average across three demographic attributes, eleven mixtures, and five post-training paths.Source: Table 21 · p. 45, §H.2 · p. 46

Related papers

Loading 2610.33371v1…