PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models

Utkarsh Agarwal, Monojit Choudhury

arXiv:2609.21094v1cs.CLcs.AI

TL;DR

The paper addresses limited evidence about how language models handle clashing moral values and whether compact models can learn reusable preferences. It builds a multilingual dilemma benchmark and tests fine-tuning and task-vector transfer. Compact models achieve high stance-adherence accuracy, while task-vector arithmetic can reverse preferences; the study’s scope is controlled forced-choice dilemmas over three principles.

  • Problem

    Direct conflicts between moral principles remain largely untested in multilingual ethics research, while model instruction-following and cross-lingual behavior can be brittle.

    Source: §1 · p. 1

  • Method

    The authors construct a multilingual benchmark of pairwise dilemmas and study compact-model fine-tuning alongside task-vector transfer that isolates preference directions from instruction components.

    Source: §1 · p. 1

  • Results

    Five-epoch LoRA adapters trained on 80 % of the benchmark achieve ≥98% accuracy on the value pairs and ≥90% on the held-out human-written gold set.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Task-vector arithmetic can flip value orderings at inference time while retaining ≥96 % of full fine-tune performance in most cases on the larger 3B model.

    Source: §7 · p. 9

    The forced-choice benchmark covers three selected principles and measures stance adherence, not general moral reasoning or explanation quality.

    Source: §6.1 · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly deployed in applications that must weigh clashing moral values, yet even strong models exhibit hidden biases and brittle instruction-following across languages. We introduce a 12,000-instance dataset of two-option dilemmas covering pairwise three value conflicts: Honesty vs. Justice, Justice vs. Autonomy, and Autonomy vs. Honesty, along with their translations into Hindi, Arabic, Spanish, and Chinese, to probe cross-lingual behavior. Benchmarking on GPT-5-mini reveals that it consistently favors Honesty over Autonomy across all five languages when no policy is given. The Llama-3.2-1/3B models exhibit strong first-option bias; however, both plain fine-tuning and Direct Preference Optimization fine-tuning effectively remove this bias, increasing accuracy to greater than 98%. In order to decouple the effect of learning correlations in the dataset from abstract values, we propose a task vector transfer based experiment where after computing the task vectors for a direction of value preference we orthogonalize it with respect to the general instruction following vector. Our experiment shows that this method is effective in isolating the direction of the specific value preference that can successfully be used to conduct task arithmetic to obtain a model with the opposite stance.

1. Introduction

The paper studies whether compact language models can learn and reversibly change preferences across three pairwise ethical conflicts. It introduces a multilingual dilemma dataset and evaluates policy adherence and task-vector transfer.

Objectives

  • The paper asks whether compact 1B–3B models can implicitly learn value preferences and whether learned preferences can be isolated and reused to change priorities without retraining.The proposed experiment orthogonalizes a value-preference task vector with respect to the general instruction-following vector.Source: §1 · p. 1

Dataset

  • The 12,000-instance dilemma dataset pairs Honesty, Justice, and Autonomy and is available in Arabic, Chinese, English, Spanish, and Hindi.Each dilemma has a unique resolution given an ethical stance or policy.Source: §1 · pp. 1–2

Policy Adherence experiments

  • PEFT-trained compact models learn value preferences beyond word-based correlations, as tested on an out-of-domain human-annotated gold dataset.The introduction reports that untrained models struggle with in-context policies and may show value or positional biases.Source: §1 · p. 2

Task-vector transfer

  • Task-vector transfer subtracts instruction-only components to isolate a preference vector, then uses scaled vector subtraction to reverse a value ordering.The method retains most of full PEFT utility, but the preference vectors are nearly orthogonal and reliable transitive composition is not observed.Source: §1 · p. 2

Contributions

  • The authors frame the work as formalizing ethical dilemmas and testing whether compact models can learn stable policies and shed position bias.They also report validation on a held-out human-authored gold set.Source: §1 · p. 2

2. Background

Prior work reports ethical steering and value biases in language models, with brittleness and cross-lingual robustness remaining concerns. This paper studies explicit stances and whether their representations can be isolated, reversed, and composed through weight manipulation.

Prior work

  • Prior studies describe both steerability and value-specific biases in frontier models, including degraded consistency in lower-resource languages.Ethical dilemmas are used to probe ethical reasoning by pitting principles against each other.Source: §2 · p. 2

Research gap

  • Prior multilingual ethics research has focused on stereotypes and toxicity, while direct conflicts between universal moral principles remain largely untested.Smaller open models’ ethical behavior and cross-lingual robustness are also described as largely unexplored.Source: §2 · p. 2

Motivation

  • The paper motivates modular alignment through value pluralism, which favors accommodating conflicting human values rather than collapsing them to one axis.Its study examines resolving dilemmas under explicit moral stances.Source: §2 · p. 2

Formalization

  • An ethical stance is a partial ordering over ethical principles, and a dilemma comprises a scenario, resolution options, and the stance the agent should follow.For the studied binary stance, one option aligns with the ordering and the other with its reversal.Source: §2 · pp. 2–3

Modularity hypotheses

  • The authors test whether a stance can be inverted by isolating a stance vector, then whether pairwise value relations can be chained transitively.Their task-vector framing treats a fine-tuned-minus-base weight difference as a vector and separates instruction and preference directions before recombination.Source: §2 · p. 3

3. Dataset

The dataset operationalizes dilemmas as conflicts between two virtuous values, with a stance determining the preferred option. It uses three selected principles, generated and validated examples, parallel translations, and a separately annotated human-written gold set.

3. Dataset

  • The dataset defines an ethical dilemma as a decision between two options, each supported by a value, where neither is objectively correct absent a stance.The stance is a strict ordering of the two principles in conflict.Source: §3 · p. 3

3.1. Ethics Principles

  • The study uses Honesty, Justice, and Autonomy to form three pairwise stances, explicitly limiting its claims to this operationalization rather than an exhaustive set of ethical principles.The principles were selected as distinct and well-defined, and the pairs are used to build training and test dilemmas.Source: §3.1 · p. 3

3.2. Dataset creation

  • The authors generated 12,000 dilemmas from 20 manually created scenarios, with 4,000 samples for each value pair and 3,200/400/400 train/dev/test splits.A validator proposed edits for 4,055 samples; experiments use the postvalidation dataset. Each sample contains a scenario/question and two value-linked options.Source: §3.2 · pp. 3–4

3.3. Translation

  • Hindi, Arabic, Spanish, and Chinese versions are translations of the English dilemmas, so the multilingual evaluation tests language transfer rather than cross-cultural equivalence.The translations were sampled for backtranslation and native-speaker inspection, and the train/test splits are parallel across languages.Source: §3.3 · p. 4

3.4. Human Gold Test Set and Annotation

  • Six peers created human-written dilemmas, and annotators assessed generated examples; the gold set retained samples agreed to be valid and relevant, with disagreements revised after discussion.Each data point was rated by two annotators, who judged appropriateness and label correctness with confidence scores.Source: §3.4 · p. 4

4. Methodology

The methodology measures unprompted value preferences and prompted stance adherence, trains compact Llama models with LoRA SFT or DPO, and tests weight-based preference reversal by separating preference from instruction-following directions.

4.1. Baseline Models

  • GPT–5-MINI is evaluated without a preference instruction for inherent value preference and with an explicit stance for accuracy; Llama-3.2 1B/3B are tested for value-conditioned choices and position bias.Table 1 reports baseline preference and Table 2 reports prompt-steering accuracy.Source: §4.1 · pp. 4–5, Table 1 · p. 5

4.2. Parameter-Efficient Fine-tuning

  • LoRA adapts only the attention query, key, and value projections, using rank 4 and α=8 for 1B models and rank 16 and α=16 for 3B models.The adapters are merged into the base weights for inference; training uses the train split of 3,200 samples.Source: §4.2 · p. 5

4.3. Direct Preference Optimization (DPO)

  • DPO compares preferred and rejected responses for each dilemma and stance, using single-token A/B completions without chain-of-thought.It uses the same language-specific splits and LoRA setup as SFT, with three seeds and reported mean ± standard deviation.Source: §4.3 · p. 5

4.4. Task Vector for Preference Alignment

  • Task-vector arithmetic starts from a model trained for one preference and aims to reverse that preference without further training.The basic procedure negates the preference task vector and adds it to the base model.Source: §4.4 · p. 5

Algorithm 1

  • Directly negating the full task vector degraded instruction-following, so the proposed method uses opposing-preference models to estimate and remove the instruction direction before reversing the preference vector.The paper reports that this orthogonalization isolates the preference vector.Source: Algorithm 1 · p. 6
  • The task-vector procedure tunes mixing weights γ1 and γ2 on the development set, using a coarse grid followed by a finer local search.The stated ranges are γ1 ∈[0, 3] and γ2 ∈[0, 2], with steps of 0.3 and then 0.1.Source: Algorithm 1 · p. 6

4.5. Inference; 4.6. Evaluation Metric

  • At inference, models receive the dilemma and an instruction to return only A or B; evaluation measures stance accuracy and tracks how often each option position is chosen.The SFT and DPO figures report gold-test accuracy by language and task.Source: §4.5 · p. 6, §4.6 · p. 6, Figure 2 · p. 6

5. Results

Task-vector transfer reverses learned value preferences while retaining much of fine-tuning performance, while SFT and DPO perform strongly on human-written dilemmas. Baselines reveal both value and position biases, and prompt steering is less reliable in Hindi.

5.3. Task-Vector for Alignment Steering

  • Task-vector steering retains ≥93% of full fine-tuning performance on BC and ≥80% on the other pairs for 1B models; 3B models retain ≥98% on two tasks and 85% on the third, in Hindi.The reported gold-set results are comparable to test-set results, and performance is resilient to the choice of instruction-vector models.Source: Figure 4 · p. 7, §5.3 · p. 7

5.1. Baselines and Prompt Steering

  • Without policy instructions, GPT–5-MINI chooses the Justice-aligned option in about 70% of Justice–Autonomy cases across languages and similarly favors Honesty over Autonomy; its Honesty-over-Justice preference is weaker.Prompt steering can reverse its baseline bias, but Hindi performance is poorer, and Task AB accuracy for Honesty over Justice falls from 64.5% to 61.8% under that policy prompt.Source: §5.1 · p. 7
  • Llama-1B and 3B models behave nearly at random at baseline and show strong position bias, often choosing the first option regardless of the intended value preference.The experiments compare zero-shot behavior, fine-tuning on a human-written gold set, and task-vector steering.Source: §5 · p. 6, §5.1 · p. 7

5.2. LoRA SFT and DPO Training

  • SFT exceeds 98% accuracy across value pairs and languages on the synthetic test set and exceeds 90% on the human-written gold set; DPO shows a comparable trend with a small drop relative to SFT.The reported performance suggests the learned stance extends beyond synthetic templates.Source: §5.2 · p. 7

6. Observations

Task-vector transfer reverses learned value orderings with high accuracy, while fine-tuning makes compact models reliably follow ethical policies. The results also show that pairwise preference directions do not compose transitively, and the evaluation remains limited to a controlled setting.

6. Observations

  • GPT–5-MINI favors Justice over Autonomy in roughly 70% of cases across languages without preference instructions.It also prefers Honesty over Autonomy and shows a weaker Honesty-over-Justice bias.Source: §6 · p. 8
  • In-context stance steering can change GPT–5-MINI’s behavior against its bias, but performance is weakest in Hindi and varies by language.For Honesty over Justice, its choice rate falls from the unsteered case despite a prompt supporting that same direction.Source: §6 · p. 8
  • Zero-shot Llama-3.2 1B and 3B models perform nearly at random and show strong positional bias, usually choosing Option 1.Arabic is an exception in which they sometimes primarily choose Option 2.Source: §6 · p. 8
  • ≥98% accuracy across tasks and languages follows five-epoch LoRA SFT or DPO fine-tuning, which eliminates position bias.On 60 human-written dilemmas, fine-tuned checkpoints largely maintain high performance and transfer to unseen narrative styles.Source: §6 · p. 8
  • ≥97% of full fine-tune accuracy is retained by 3B task-vector transfer across tasks and languages except AB with Hindi.The 1B model also retains high performance, though its AB and CA results are worse; optimal transfer uses moderate instruction and preference scaling.Source: §6 · p. 8
  • Preference-vector composition for transitivity yields near-random results, and different stance vectors have angles above 80°.Preference reversal therefore supports local binary stance changes, not a fully compositional linear geometry of values.Source: §6 · p. 8

6.1. Limitations

  • The benchmark tests forced-choice adherence to three selected principles, not general moral reasoning or explanation quality.Most examples are generated and machine-translated; the human-written gold set is small, and experiments cover only Llama-3.2 1B and 3B. Task-vector transfer also requires preference-finetuned checkpoints and dev-set tuning of mixing coefficients.Source: §6.1 · p. 8

7. Conclusion

The paper presents a multilingual benchmark of ethical dilemmas and finds that compact Llama-3.2 models can learn policies that generalize to human-written dilemmas. Its task-vector approach can reverse value orderings without retraining, while the authors call for broader human and cultural validation.

7. Conclusion

  • The 12,000-dilemma benchmark covers three value pairs and uses LLM-based validation, a small human audit, and a separate human-written gold set..Source: §7 · p. 8
  • ≥98% accuracy on value pairs and ≥90% on a held-out human-written gold set are achieved by five-epoch LoRA adapters trained on 80% of the benchmark.The authors report that the gold-set result generalizes beyond synthetic wording and does not rely on surface heuristics.Source: §7 · p. 8
  • Prompting improves stance adherence but remains uneven across value pairs and languages, while PEFT is required for robust policy shifts.The reported weaknesses are especially marked for Hindi and cases opposing the model’s default tendencies.Source: §7 · p. 8
  • Task-vector arithmetic flips value orderings without retraining, retaining at least 96% of full fine-tune performance in most cases on the 3B model.The conclusion describes an isolatable preference direction after instruction components are subtracted, enabling on-demand stance changes without maintaining multiple aligned copies.Source: §7 · p. 9
  • A separately written gold set and a small human audit provide evidence of transfer beyond the original generation templates, while broader human and cultural validation remains needed..Source: §7 · p. 9

A. Prompt Structure for Evaluation

The evaluation prompt presents a situation, asks a decision question, and offers two alternative resolutions.

A. Prompt Structure for Evaluation

B. Scenarios for the Dilemmas

The scenario list spans interpersonal, social, professional, and global contexts, including relationships, community life, and public settings.

B. Scenarios for the Dilemmas

D. Plots

The plots show test accuracy for SFT and DPO models, a scatter of optimized coefficients, and task-vector transfer efficiency relative to full LoRA fine-tuning.

D. Plots

  • Figure 5 reports SFT test accuracy by language and task, with each model trained on the corresponding training data.Source: Figure 5 · p. 12
  • Figure 6 reports DPO accuracy for each language-task pair, with three training seeds per pair.Source: Figure 6 · p. 12
  • Figure 7 plots the optimum γ1(γinstr) and γ2(γpref), showing the top 80% of points scaled by test accuracy and jittered to prevent overlap.Source: Figure 7 · p. 13
  • Figure 8 expresses task-vector accuracy as a percentage of full LoRA accuracy; labels identify the reversed preference and, in boxes, instruction-vector models, with the better combination above.Source: Figure 8 · p. 13

Related papers

Loading 2609.21094v1…