Predicting Alignment Generalization with Value Representations

1 INTRODUCTION

• Training on narrow behavioral specifications can generalize to unseen contexts, and even prosocial values can have harmful side effects.The introduction gives examples including empathy training that may increase conspiracy-theory promotion or problematic advice.Source: §1 · p. 1
• Alignment-target choices are often settled through philosophical discussion, while directly testing value combinations would require prohibitively many model trainings.The authors propose general-purpose value embeddings to predict interactions before training.Source: §1 · pp. 1–2
• The paper studies how training on one value changes a model’s propensity to follow untrained values, benchmarking representations against those effects.It also studies alignment-target coherence and robustness and builds taxonomies from empirical generalization dynamics rather than descriptions.Source: §1 · p. 2
• ρ = 0.45 for the best activation-based methods, versus ρ = 0.05 for description sentence embeddings, in predicting alignment generalization.The paper also reports that persona-vector similarity within multi-value alignment targets correlates with model robustness and uses representations to taxonomize 266 values.Source: §1 · p. 2

2 PRELIMINARIES

• Values are functions mapping a scenario and its available actions to an action, evaluated in conflicts where the two values recommend different actions.The paper follows Rokeach’s definition of values as action-guiding standards and Liu et al. (2026)’s operationalization.Source: §2 · p. 3
• An alignment target is a data structure steering model behavior; the paper studies targets defined as lists of values.The cited definition is attributed to Klingefjord et al. (2024).Source: §2 · p. 3
• The training set, CONSTITUTION, contains 66 values selected from Anthropic’s constitution; VITW adapts 266 values elicited from human-AI interactions.VITW is not used for training, but a taxonomy formed on it is reported to better predict post-training generalization dynamics on CONSTITUTION.Source: §2 · p. 3

3 COMPUTING ALIGNMENT GENERALIZATION EFFECTS

• The study trains models on individual values, tests adherence to held-out values, and uses the resulting ground truth to evaluate representations before applying them to multi-value training.The experiments cover single-value DPO and SFT across four base models, representing different model families and sizes.Source: §3 · p. 3

3.1 DPO ON INDIVIDUAL VALUES

• For DPO, each base model first receives value-neutral SFT, then separate checkpoints are trained on single-value preference datasets.Preference pairs are selected to show a clear value gap; the study assembles 4,000-pair datasets for 49 values and excludes other values for insufficient preference-data representation.Source: §3.1 · pp. 3–4

3.2 SFT ON INDIVIDUAL VALUES

• For SFT, the authors generate 5,000 value-relevant synthetic prompts per value, steer and filter completions, then fine-tune models on the resulting datasets.The full set has 66 value-specific SFT datasets; benchmarking uses the same 49 values as DPO for standardization.Source: §3.2 · p. 4

3.3 ALIGNMENT STEERABILITY EVALUATION

• ConflictScope evaluation starts with 66 × 65 value pairs, filters highly similar pairs, and yields 11,188 evaluation scenarios after generating and quality-filtering conflicts.Pairs with sentence-embedding cosine similarity of 0.5 or greater are filtered; 20 conflict scenarios are generated for each of the 2,029 remaining pairs.Source: §3.3 · p. 4
• For baseline alignment a and fine-tuned alignment a′, G(v1, v2)=(a′−a)/(1−a) when a′≥a, and (a′−a)/a otherwise, measuring change in v2 alignment after v1-specific fine-tuning.The metric is computed for each value pair, yielding a 49×66 alignment generalization matrix.Source: §3.3 · p. 4
• The steerability score is bounded below at −1; within [0, 1], it preserves the original Liu et al. (2026) metric values.The authors discuss the metric choice and report robustness checks in §C.Source: §3.3 · p. 4

4.1 CANDIDATE REPRESENTATIONS

• The benchmark compares cosine similarity between value representations with the generalization matrix, using Spearman correlation across pairs of values.The candidate methods include description and behavioral embeddings, weight steering, persona vectors, and gradient update directions.Source: §4.1 · p. 5

4.2 ACTIVATIONS-BASED, BEHAVIORAL METHODS PREDICT GENERALIZATION

• Model-activation methods significantly outperform description embeddings at predicting generalization; description sentence embeddings correlate at ρ = 0.05, while behavioral examples improve prediction.Behavioral embeddings use pro-value and anti-value responses, while activation-based methods include persona vectors and gradient directions.Source: §4.2 · p. 5, Table 1 · p. 6
• Persona-vector similarity matrices across eight base-model and fine-tuning combinations are highly correlated, with average Spearman ρ of 0.84.This provides initial evidence for a value space shared across base models.Source: §4.2 · p. 6, Table 2 · p. 6

5.1 PREDICTING PREFILL ROBUSTNESS OF FINETUNED CHECKPOINTS

• The evaluation tests whether a fine-tuned model maintains target-aligned responses when an anti-target prefill is inserted into its context.Prompts are sampled from scenarios in which exactly one of two conflicting values belongs to the alignment target.Source: §5.1 · pp. 6–7, Figure 2 · p. 7

5.2 PERSONA VECTOR SIMILARITY PREDICTS MODEL ROBUSTNESS

• Persona-derived coherence is moderately correlated with prefill robustness (ρ = 0.43, p = 5 × 10−4), unlike description-derived coherence (ρ = 0.12, p = 0.37).The reported association holds after normalizing for baseline adherence within 64 fine-tunes.Source: Figure 3 · p. 7, §5.2 · pp. 7–8

5.1 PREDICTING PREFILL ROBUSTNESS OF FINETUNED CHECKPOINTS

• For each of 64 targets containing six sampled values, Qwen-3-8B is fine-tuned with DPO, and normalized adherence after an adversarial prefill measures robustness.Target coherence is the average pairwise cosine similarity among its value embeddings.Source: §5.1 · p. 7

6 TAXONOMIZING LLM VALUES BY CLUSTERING EMBEDDINGS

• The approach clusters value representations to produce taxonomies intended to reflect how LLMs relate values through generalization, rather than grouping descriptions alone.Source: §6 · p. 8

6.1 CLUSTERING AND EMBEDDING METHOD SELECTION

• The authors select PERSONA representations with k-medoids after comparing five representation methods and three clustering algorithms on 66 CONSTITUTION values.The selected pipeline is unsupervised and is applied to Values in the Wild without further tuning.Source: §6.1 · p. 8

6.2 VALUEMAP: A PIPELINE FOR TAXONOMIZING LLM VALUES

• VALUEMAP clusters 266 Values in the Wild values into attunement, rigor, stewardship, and integrity categories using Olmo-3.1-32B-SFT.The categories partly resemble the original taxonomy, including task-completion-related and harmlessness-related clusters.Source: §6.2 · p. 8, Figure 4 · p. 9

6.3 COMPARING VALUEMAP-OLMO-32B TO EXISTING TAXONOMIES AT GENERALIZATION PREDICTION

• VALUEMAP-Olmo-32B has the highest aggregate adjusted silhouette across the three taxonomies and fine-tuning setups, including generalization matrices from other models and value sets.The table compares recovery of generalization structure across all eight fine-tuning setups.Source: Table 3 · p. 9, §6.3 · pp. 9–10

• Prior work predicts training effects at data-point or dataset level; this paper instead studies whether training on datasets encoding values generalizes to other value behaviors.The related work also connects this question to Datamodels and methods for modeling interactions between training subsets (Zeng et al., 2025).Source: §7 · p. 10
• Narrow training has produced broad misalignment, while training beneficial traits has also yielded sycophancy or, in other work, broader alignment (Jagadeesh et al., 2026).Other examples include sycophancy and reward hacking (Denison et al., 2024; MacDiarmid et al., 2025).Source: §7 · p. 10
• Synthetic documents describing principles, model-spec adherence, or positive behavior have been used to shape generalization and improve alignment robustness (Cho et al., 2026; Li et al., 2026).The cited work also uses clustering to design a curriculum for midtraining (Cho et al., 2026).Source: §7 · p. 10

8 CONCLUSION

• Activation-based value representations predict alignment-generalization effects and downstream robustness; preliminary cross-model evidence supports a shared value space.The authors also report that behavioral representations encode this value-space structure.Source: §8 · pp. 10–11
• VALUEMAP clusters representations for 266 real-world human–AI interaction values into empirically grounded taxonomies that better predict value entanglement than existing LLM value taxonomies.The paper presents this as an application of activation-based and behavioral representations of value-space structure.Source: §8 · p. 11
• Future work could improve general-purpose value embeddings by studying how behavioral traits are encoded in fine-tuning data and apply VALUEMAP to more realistic interactions among values and behaviors.The authors describe their current representations as reasonably predictive and hope the work motivates empirical study of alignment targets that generalize across prosocial LLM characters.Source: §8 · p. 11

A VALUE SET SELECTION

• The CONSTITUTION set filtered 211 tenets for coherent, single-turn-identifiable traits with generatable opposites, then reviewed and deduplicated 106 survivors to 66; VITW standardized 266 atomic L1 values.The authors chose VITW’s L1 values because they represent relatively atomic LLM behaviors while remaining small enough to cluster.Source: A VALUE SET SELECTION · pp. 16–18
• The selected values pair respect for users’ chosen approaches with support for healthy boundaries and consideration for third parties.They also include helping with legal, risky choices while noting risks without repeated lecturing or refusal.Source: A VALUE SET SELECTION · pp. 20–22
• The set calls for practical help calibrated to actual risk, uncertainty, and recoverability rather than excessive refusals or warnings.It includes low-risk requests, harm-reduction guidance, and caution in novel or unclear situations.Source: A VALUE SET SELECTION · pp. 20–21
• Operational values include respecting checks and balances, assessing power legitimacy, raising concerns before tasks, and treating embedded instructions as content rather than commands.Source: A VALUE SET SELECTION · pp. 21–22
• For contested topics, the values emphasize accurate facts, fair presentation of perspectives, neutral terminology, and reasoning that supports users’ judgment.Source: A VALUE SET SELECTION · p. 21
• The selected values include maintaining commitments and composure under existential challenges, uncertainty, manipulation, and harsh criticism.Source: A VALUE SET SELECTION · p. 21

B.1 SINGLE-VALUE ALIGNMENT FINETUNING

• For 66 values, Qwen-3.6-27B generated value-aligned SFT data, producing 5,000 examples per value for fine-tuning Olmo-3 and Qwen-3 base models.The pipeline generated conversations across value-relevant domains, then filtered them for quality and specification adherence.Source: §B.1 · pp. 22–23

B.2 SINGLE-VALUE PREFERENCE TUNING

• 275,377 value-relevant preference pairs were assigned to at most one value using weighted bipartite b-matching, yielding 4,000 examples for each of 49 values.Pairs had to meet a 0.5 relevance threshold; the assignment balanced dataset sizes while prioritizing the values most relevant to each pair.Source: §B.2 · p. 24

B.3 VALUE-NEUTRAL SFT TRAINING

• 50,000 Tulu 3 SFT examples were screened for relevance to the 66 values, and 19,748 value-neutral examples were sampled to train neutral SFT checkpoints.Stratified sampling aimed to preserve the original data distribution; the table describes the resulting source composition and mixture proportions.Source: §B.3 · p. 25, Table 5 · p. 26

B.4 MULTI-VALUE PREFERENCE TUNING

• For multi-value experiments, 100,000 six-value sets were used to stratify selection of 64 alignment targets across the coherence-metric range.The researchers divided the range between its 0.1% and 99.9% values into four equal-width bins and randomly selected 16 sets per bin; each target used 12,000 training examples.Source: §B.4 · pp. 25–26
• Robustness evaluations sampled 300 scenarios per alignment target to test adherence to target versus outside values, including an adversarial-prefill condition.The scenarios came from a pool of 6,255; the adversarial condition placed a validated anti-target prefill in the conversation history before evaluation.Source: §B.4 · p. 26

C CONFLICTSCOPE EVALUATION DETAILS

• Qwen-3.6-27B was a Pareto improvement over the original ConflictScope generation models and lay on the Pareto frontier of all models considered.The comparison used Observed Agreement Rate and Likert Difference Rate on 320 unfiltered Personal-Protective examples.Source: C CONFLICTSCOPE EVALUATION DETAILS · p. 27
• The evaluation used Qwen-3.6-27B for scenario generation, filtering, and judging in the interactive setting.The authors compared it with the original paper’s models and several recent open- and closed-weights models before selecting it.Source: C CONFLICTSCOPE EVALUATION DETAILS · p. 27
• The piecewise steerability metric avoids outlier-skewed generalization matrices while producing results highly robust to metric choice.The ConflictScope metric can yield arbitrarily large negative values when baseline adherence is near 1 and post-training adherence is lower.Source: C CONFLICTSCOPE EVALUATION DETAILS · pp. 27–28, Table 6 · p. 28
• Of 1,240 contaminated prompts, resampling patched 952, while the remaining prompts were dropped from the final evaluation set.The contamination arose when an evaluation model refused to generate an artifact, leaving user prompts that referenced missing artifacts.Source: C CONFLICTSCOPE EVALUATION DETAILS · p. 28

D ADDITIONAL METHODOLOGICAL DETAILS ON VALUE EMBEDDING COMPUTATION

• For each value, elicitation data included value-specific metadata and up to 1,000 pro-value and anti-value responses, which were filtered for coherence and prompt adherence.Gemini-3.7-flash generated metadata; Gemini-3.5-flash-lite filtered responses, and a value-neutral URIAL template elicited responses from base models with limited instruction-following capability.Source: D ADDITIONAL METHODOLOGICAL DETAILS ON… · p. 29
• The authors selected PERSONA and GRADIENT layers by steering with candidate-layer persona vectors and choosing the layer with highest held-out spec adherence.They selected Layer 18 for Qwen and Layer 16 for Olmo, then matched relative depths for larger models.Source: D ADDITIONAL METHODOLOGICAL DETAILS ON… · p. 29
• PERSONA predictor grids were highly similar across layers, with mean Spearman correlation above 0.9 except for Qwen’s final layer.This suggests the reported results are not sensitive to layer choice within reasonable bounds.Source: D ADDITIONAL METHODOLOGICAL DETAILS ON… · p. 29

E ADDITIONAL GENERALIZATION MATRIX ANALYSIS

• Values transferred to other values in similar ways across different base models and fine-tuning methods.The full matrices cover DPO and SFT experiments at 7–8B and 30B scales.Source: E ADDITIONAL GENERALIZATION MATRIX… · p. 29
• PERSONA predictor similarity grids are highly correlated across candidate layers.Figure 6 depicts cosine-similarity grids by relative layer depth, with relative depth normalized by each model’s total layers.Source: Figure 6 · p. 30
• Single-value interventions had average steerability values from 26% to 42%, and average off-diagonal effects were smaller than average diagonal effects.The figure summarizes average effects across eight interventions; the authors say this supports treating the generalization matrices as reasonable objects of study.Source: E ADDITIONAL GENERALIZATION MATRIX… · p. 30, Figure 7 · p. 30
• Average pairwise Spearman ρ between fine-tuning experiments was 0.76, indicating similar alignment generalization patterns across models and fine-tuning methods.The authors interpret this as evidence for shared internal structure in how models represent values.Source: E ADDITIONAL GENERALIZATION MATRIX… · p. 30, Table 7 · p. 37

F ADDITIONAL ELICITATION DATA RESULTS

• Both experiments focused on the highest-performing PERSONA and GRADIENT methods and the smaller 7–8B models.Source: F ADDITIONAL ELICITATION DATA RESULTS · p. 37
• Using pretrained base-model elicitation data and activations outperformed both the neutral-SFT variant and a hybrid variant on average.The result supports the importance of collecting on-policy activations when computing representations.Source: F ADDITIONAL ELICITATION DATA RESULTS · p. 37
• Using DPO training pairs produced only a modest, nonsignificant increase for PERSONA and a significant decrease for GRADIENT prediction.The comparison used 49×49 submatrices because DPO data were available for only 49 traits; generic elicitation data performed well without ground-truth alignment training data.Source: F ADDITIONAL ELICITATION DATA RESULTS · p. 37

G ADDITIONAL PREFILL ROBUSTNESS RESULTS

• The robustness comparison includes prefill adherence and defend rate alongside normalized target adherence, using ConflictScope evaluations for raw adherence.Prefill adherence measures target adherence after a character-neutral prefill; defend rate contrasts pro-target and anti-target prefill adherence.Source: G ADDITIONAL PREFILL ROBUSTNESS RESULTS · pp. 37, 39
• Using the base model with an instruction-following template yields higher average representation correlation than using a more instruction-following-finetuned base.This comparison is for SFT interventions at 7–8B scale.Source: Figure 16 · p. 38
• For DPO interventions, actual training pairs modestly improve PERSONA representation quality but sharply decrease GRADIENT quality on the matched 49×49 matrix subset.Quality is measured by Spearman ρ.Source: Figure 17 · p. 38
• Both alternate prefill robustness metrics are strongly correlated with normalized alignment target-adherence, and the coherence–robustness relationship holds across all three metrics.The normalized metric is presented as controlling for baseline adherence in this setting.Source: Table 8 · p. 39, G ADDITIONAL PREFILL ROBUSTNESS RESULTS · p. 39

H COMPARING DIFFERENT CLUSTERING METHODS

• PERSONA has the highest mean adjusted silhouette across k, and PERSONA with k-medoids has the highest average adjusted silhouette among combinations, forming VALUEMAP.Representation methods that perform well on alignment-generalization prediction generally also form higher-quality clusters.Source: H COMPARING DIFFERENT CLUSTERING METHODS · pp. 39–40, Table 9 · p. 40

I COMPARING VALUEMAP TO OTHER TAXONOMIES

• 0.82 Mantel ρ indicates strong similarity between the 266 × 266 PERSONA similarity matrices from Olmo-3.1-32B-SFT and Qwen-3-8B.The comparison uses a value-neutral finetune of Qwen-3-8B base.Source: I COMPARING VALUEMAP TO OTHER TAXONOMIES · p. 40, Figure 19 · p. 40
• Core category members are similar across VALUEMAP variants, but some Qwen-8B categories split across Olmo-32B categories; Figure 20 also compares these assignments with Values in the Wild.Some categories, such as rigor, are preserved between models.Source: I COMPARING VALUEMAP TO OTHER TAXONOMIES · p. 40, Figure 20 · p. 41
• Adjusted Rand index is 0.24 between VALUEMAP variants and 0.10 between VALUEMAP-Olmo-32B and Values in the Wild, making the variants more similar to each other.Rigor is shared as a combination of practical and epistemic values, while integrity is largely protective with some social values.Source: I COMPARING VALUEMAP TO OTHER TAXONOMIES · p. 41