Source-linked AI summary
Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversations
Enrico Liscio, Anne Arzberger, Aditya Shankar, Pradeep K. Murukannaiah
TL;DR
Value orientations vary by person and context, while most inference-time steering evidence comes from single-turn settings. The paper evaluates four steering methods in multi-turn conversations with opposing or reinforcing user and assistant value pressure, finding that outcomes often track the assistant model’s baseline bias and can fade over turns.
Problem
Value preferences vary across people and contexts, while inference-time value steering has been evaluated primarily on single-turn responses rather than evolving conversations with competing user pressure.
Source: §1 · p. 1Method
The study compares four prompt- and activation-based steering methods across three models and two Schwartz value trade-offs, tracking value alignment with a linear probe during multi-turn conversations.
Source: §1 · p. 2, §4.1 · p. 4Results
In most opposing conditions, the side matching the model’s baseline bias prevails, and apparent early steering success often fades or reverses over subsequent turns.
Source: §6 · p. 8Takeaways & Limitations
Value steering should be evaluated over full conversations and in both directions of a value trade-off.
Source: §7 · p. 8The study uses one simulated user model and ten-turn conversations, so results may differ with more varied users or longer, topic-shifting interactions.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Human values shape how we expect language models to behave. Dominant alignment approaches treat values as global objectives, although the appropriate value orientation is personal and contextual. Training-free alignment can steer models toward target values at inference time, yet has been largely evaluated in single-turn settings and with a single steering direction (e.g., toward safety). We evaluate four prompt- and activation-based steering methods in multi-turn conversations, where user and assistant models push toward the same or opposing, equally desirable values. We track alignment with a linear probe across three models and two value trade-offs. When assistant and user pull in opposite directions, the side matching the assistant's baseline bias often prevails, and effects that look successful early often fade or reverse after a few turns. Steering should therefore be evaluated over full conversations and in both directions of a value trade-off.
1 Introduction
Value alignment must account for values that vary across people and contexts, challenging fixed training-time objectives. This paper tests whether inference-time steering holds across multi-turn conversations when users and assistants reinforce or oppose one another.
1 Introduction
- Fixed training objectives aggregate preferences, but value preferences vary across individuals and contexts, challenging the assumption that one objective captures the full range of user values.The paper identifies this as a challenge for approaches such as RLHF and DPO.Source: §1 · p. 1
- Prior inference-time value steering has focused mainly on single-turn responses and on moving away from an undesired behavior toward one desirable target.Competing desirable orientations, such as autonomy and respect for traditions, may both be appropriate depending on the user and context.Source: §1 · p. 1
- The study compares four prompt- and activation-based steering strategies across three models, ten Schwartz-grounded scenarios, and two value trade-offs, tracking value expression across turns.Users and assistants are set to promote either the same or opposing values, and a probe tracks value-relevant directions in activation space.Source: §1 · p. 2
- When user and assistant pull in opposite directions, the side matching the model’s intrinsic bias prevails in most cases, while apparent early success often fades or reverses by the final turns.Activation-based steering shows less reversion but can break conversation coherence in a direction-specific way.Source: §1 · p. 2
2 Related Works
The related work frames value alignment as contextual because value orientations can conflict without a unique ground truth. Existing training-free and multi-turn evaluations leave open how value steering behaves when users and assistants pull toward either side of a trade-off.
2.1 Aligning LLMs to Human Values
- Value orientations are pluralistic, conflicting, and dependent on individuals and context, so value alignment has no unique ground truth.Schwartz Value Theory represents ten values in a circumplex, with opposing values forming trade-offs between equally desirable orientations.Source: §2.1 · p. 2
- Fine-tuning on value-annotated data can shift model value profiles, but fixes the profile at training time rather than adapting it to changing orientations.This motivates approaches that adjust model behavior at inference time.Source: §2.1 · p. 2
2.2 Training-Free and Multi-Turn Alignment
- Training-free alignment methods span prompts, input rewriting, activations, and logits, but evidence for these methods is largely based on single-turn responses.Multi-turn evaluations commonly assess behavioral consistency or persona retention rather than shifts in value orientation.Source: §2.2 · p. 2
- Unlike prior multi-turn steering toward one desirable target, this study steers toward either pole of a value trade-off and steers the user as well as the assistant.A linear probe tracks internal value representations across turns, including orientations that outputs may not express.Source: §2.2 · p. 2
3 Evaluating Value Alignment
The evaluation stages assistant–user conversations with controlled value pressure and compares prompt- and activation-based steering across models. Scenarios, steering conditions, and interventions are designed to test reinforcing and opposing value directions.
3.1 Conversational Scenarios
- Ten conversational scenarios represent self-direction↔security and universalism↔power, with five scenarios per trade-off and a manually specified opening user utterance for each.The user model is instructed with the value tension and context to continue each conversation.Source: §3.1 · p. 3
3.2 Assistant Models
- The evaluation uses Qwen3-14B, Llama-3.1-8B-Instruct, and Llama-3.1-70B-Instruct as assistant models.The models span multiple families and sizes.Source: §3.2 · p. 3
3.3 Steering Methods
- Four inference-time methods pair system prompting and Contrastive Activation Addition with adaptive counterparts that update steering as the conversation unfolds.CAA adds a steering vector derived from activation differences between examples labeled with opposing values; ASP injects corrective prompts when drift is detected, while SCAA adjusts steering strength based on the current activation’s alignment with the steering direction.Source: §3.3 · p. 3
3.4 Experimental Design and Setup
- Seven conditions vary whether assistant and user value directions reinforce or oppose one another, across 570 configurations repeated with five seeds.Conversations are limited to ten turns; CAA and SCAA use the smallest steering strength that produces clear effects because larger values deteriorate generation quality.Source: §3.4 · pp. 3–4
4 Measuring Value Alignment
The paper uses a linear probe to track Schwartz-value representations in model activations during generation, then aggregates alignment trajectories and assesses response quality. Its measures capture both average steering effects and changes between the first and last turns.
4.1 Value Alignment Tracking Probe
- A linear probe maps model activations to Schwartz values and estimates value representations at each conversational turn.The probe uses the residual-stream activation of the last input token at a selected layer to produce ten value scores.Source: §4.1 · p. 4
- The probe combines three Schwartz-annotated datasets into a harmonized ten-value corpus and is trained separately for each tested model.The best-layer probes reach AUROC of 0.82–0.84, Spearman correlations of 0.65–0.66, and F1 around 0.6.Source: §4.1 · p. 4
4.2 Evaluation Metrics
- Relative alignment is the difference between standardized probe scores for the two values in the intended trade-off, tracked across conversation turns.Temporal Effect aggregates average alignment shift, while Net Drift captures the change from the first to the last turn.Source: §4.2 · p. 4
- Generation quality is assessed through response coherence and repetitiveness, alongside alignment trajectories across turns.An LLM judge scores coherence on a 0–100 scale; repetitiveness is the fraction of sentences with SBERT cosine similarity above 0.8 to a prior-turn sentence.Source: §4.2 · pp. 4–5
5 Results
Across three models and two value trade-offs, the model’s baseline bias often determines which side prevails under opposing pressure. Conversational trajectories also reveal drift that averages can hide, while steering methods differ in coherence failures.
5.1 Direction and Magnitude of Alignment
- When only the user was steered, the assistant tended to follow; adding reinforcing assistant steering changed little for Qwen and Llama-8B.Llama-70B was a partial exception: SP and ASP added shifts beyond user-only conditions in both trade-offs.Source: §5.1 · p. 6
- The baseline-favored value prevailed in 39 of 48 opposing-steering cases, regardless of whether the assistant or user pushed toward it.Exceptions were spread across methods.Source: §5.1 · p. 6
5.2 Conversational Drift
- 0.97 in universalism↔power and −0.87 in self-direction↔security were Qwen’s unsteered net drifts, despite positive average TE in the latter trade-off.Trajectories show initially strong value expression declining toward a milder level aligned with the user pull.Source: §5.2 · p. 6, Figure 2 · p. 6
- Under AbUb, CAA and SCAA showed the least reversion, while SP crossed zero by turn 7 toward universalism and ASP stayed on the power side.Qwen was an exception in self-direction↔security, where no method reverted.Source: §5.2 · pp. 6–7, Figure 3 · p. 7
- In AaUb for self-direction↔security, positive average TE coexisted with ND of −1.96 for Qwen and −0.73 for Llama-70B.Qwen’s value expression fell from A1 = 2.10 to 0.1–0.4 by turn 2, showing how an early effect can recede.Source: §5.2 · p. 7, Figure 4 · p. 7
5.3 Conversation Quality
- SP and ASP stayed near BASE coherence, while CAA and SCAA collapsed into repetitive looping in direction-specific patterns that varied by model and trade-off.For self-direction↔security, CAA and SCAA collapsed when steered toward security; the universalism↔power pattern varied across models.Source: §5.3 · pp. 7–8, Table 4 · p. 7
6 Discussion
The discussion argues that baseline bias often outweighs competing steering and that successful-looking averages can conceal decay. Steering methods trade value stability against fluent generation, and their outcomes depend on steering direction.
6 Discussion
- When user and assistant pull in opposite directions, the baseline-favored value prevails in most cases, without either role winning systematically.The bias is stronger for self-direction↔security than for universalism↔power across all three models.Source: §6 · p. 8
- Single-turn snapshots and conversation averages can report success even when steering fades before the conversation ends.Only trajectory-level evaluations reveal this decay.Source: §6 · p. 8
- SP and ASP remain fluent but can drift under bias-incongruent steering, whereas CAA and SCAA show less reversion but can collapse into repetitive looping.Adaptivity delays but does not remove these failures: ASP did no better than SP in self-direction↔security, and SCAA delays CAA’s collapse.Source: §6 · p. 8
- Each method’s outcome depends on steering direction: the same method may revert fluently in one direction and loop in the other.The authors conclude that both value directions must be tested.Source: §6 · p. 8
7 Conclusions
Across multi-turn conversations, steering often loses or reverses early gains when user and assistant push toward opposing values, with the model’s baseline bias often prevailing. The study compares four steering methods across models and value trade-offs, and recommends evaluating full conversations in both steering directions.
7 Conclusions
- The study evaluates four steering methods across three models and two value trade-offs, with a simulated user pulling toward the same or opposing value.It tracks value alignment across multi-turn conversations.Source: §7 · p. 8
- When user and assistant pull toward opposing values, the side matching the model’s baseline bias prevails in most conditions, and apparent early success often fades or reverses.Whether steering succeeds also depends on its direction.Source: §7 · p. 8
- Future evaluations should cover more value trade-offs, more diverse users including real people, and longer conversations.The adaptive methods respond only to alignment drift; a multi-objective target could also balance alignment against baseline bias and generation quality.Source: §7 · p. 8
Limitations
The evaluation covers two value trade-offs and English scenarios, but the authors caution that baseline-bias differences and several design choices constrain how far the findings generalize.
Limitations
- The study uses two value trade-offs with five scenarios each; because baseline bias differs across trade-offs, its findings may not extend to other Schwartz values or models with different prior strengths.The scenarios are in English, so extension to other languages, including underrepresented ones, remains to be investigated.Source: Limitations · p. 9
- A single user model and ten-turn conversations limit inference about other users and longer, potentially topic-shifting real conversations.Absolute scores depend on the user-model choice, so conclusions rely on comparisons across conditions and methods relative to baseline.Source: Limitations · p. 9
- Value measurement uses a moderately performing probe trained on standalone text and applied to conversational activations, without validation against human judgments.Because the reported score is the difference between two value scores, the authors cannot tell which value changed or how.Source: Limitations · p. 9
- Activation-based methods use one steering coefficient per model, selected from a small sweep in which stronger steering consistently increases generation degradation.A denser sweep or per-method layer selection might reveal better operating points, though optimization is not the study’s aim.Source: Limitations · p. 9
Ethical Considerations
The paper treats both poles of each value trade-off as legitimate and does not endorse the evaluated models’ baseline preferences. It also warns that value steering could be used covertly, while emphasizing that simulated conversations do not establish effects on real users.
Ethical Considerations
- The study takes no position on which value orientation a model should express; observed baseline preferences are properties of the evaluated models, not endorsed targets.Schwartz’s framework and the scenarios do not capture the full diversity of human values across cultures.Source: Ethical Considerations · p. 9
- Deployed steering could covertly nudge users or make assistants more compliant with harmful user pressure, so the authors encourage transparency about its use.The observed fading or reversion of effects is not a safeguard, since some methods held direction against competing pressure and simulated user pressure could be adversarial.Source: Ethical Considerations · p. 9
- All conversations are model-generated, so the findings do not establish how real users would be affected; probe datasets may also carry source annotation biases and cultural skews.Source: Ethical Considerations · p. 9
A Experimental Setup
The experimental setup specifies the conversational scenarios, adaptive-prompt rubric, models and datasets, computational procedures, and value-measurement approach.
A Experimental Setup
- The setup documents scenario design, the adaptive-system-prompt judging rubric, and the data, models, hardware, and software used.Source: A Experimental Setup · p. 13
A.1 Conversational Scenarios
- Ten scenarios instantiate self-direction↔security and universalism↔power within Schwartz’s circumplex, covering varied contexts and tensions between personal choice, stability, influence, and others’ welfare.Self-direction↔security spans life domains and personal risk; universalism↔power gives weight to responsibilities and consequences beyond the decision-maker.Source: §A.1 · p. 13, Figure A1 · p. 13
A.2 Adaptive System Prompt Rubric
- The adaptive evaluator scores each value separately from −2 to 2 and, when drift is detected, inserts target- and opposing-value guidance with the evaluator’s rationale.Source: §A.2 · p. 14
A.3 Artifacts Used
- The assistant models are Qwen3-14B and Llama-3.1-8B/70B-Instruct; Gemma-4-31B serves as user model, and three existing English value-annotated datasets train the probe.The datasets are ValueNet, ValueEval, and FULCRA; no additional data were collected.Source: §A.3 · pp. 14–15
A.4 Computing and software infrastructure
- Each conversation is generated five times with five seeds; generation, including the steering-strength sweep, took 200 hours for Qwen, 50 for Llama-8B, and 96 for Llama-70B.The experiments saved dataset activations, trained probes across selected layers, and generated conversations using an NVIDIA A40 for Qwen and Llama-8B and an RTX PRO 6000 for Llama-70B.Source: §A.4 · p. 15
B.1 Probe Training and Evaluation
The probe combines harmonized Schwartz-value annotations and selects model-specific layers, where its value predictions are above chance for all four values used in the scenarios. The appendices also describe the coherence-judging rubric and list additional steering analyses.
B.1 Probe Training and Evaluation
- Probe targets harmonize ValueNet, ValueEval, and FULCRA into ten Schwartz-value dimensions, combining binary crossentropy and continuous-label Huber losses with balanced dataset sampling.Binary annotations come from ValueNet and ValueEval; continuous annotations come from FULCRA.Source: §B.1 · p. 16
- Performance rises sharply in early layers and then largely plateaus in middle-to-late layers, while MAE continues decreasing toward the final layers.For Llama-70B, AUROC rises from 0.54 to 0.79 between layers 2 and 16; AUROC, F1, and ρ then vary by at most 0.04 across middle-to-late layers.Source: §B.1 · p. 16, Table B1 · p. 16
- All four scenario values are detected above chance by every probe (AUROC 0.73–0.85), although power has the lowest ρ among the four selected values (0.24–0.30).Achievement, a neighboring value, performs worse on ρ (≤0.07).Source: §B.1 · pp. 16–17
B.2 Coherence Evaluation Prompts
- The coherence rubric evaluates response form—including logical structure, grammatical correctness, and flow—rather than helpfulness, correctness, or appropriateness of content.It allows short, well-formed responses to score highly and does not penalize brevity.Source: §B.2 · p. 17
C Additional Results
- The appendix reports a steering-coefficient sweep for CAA and SCAA, an additional alignment metric, and repetitiveness scores.The coefficient sweep reports TE and coherence.Source: C Additional Results · p. 18
C.1 Steering Coefficient Sweep
The coefficient sweep shows that stronger steering can degrade coherence without reliably increasing alignment shifts, and no single coefficient is best across directions. Additional analyses distinguish genuine alignment stability from low variation caused by collapsed, repetitive generation.
C.1 Steering Coefficient Sweep
- For Llama-8B in self-direction↔security, CAA under AbUb coherence falls from 90.3 at α = 3 to 11.6 at α = 5, while larger coefficients do not reliably produce larger alignment shifts.For Qwen, the largest CAA shift in universalism↔power occurs at α = 3 (−0.75), versus −0.36 at α = 5, where coherence is 26.0.Source: §C.1 · p. 18, Table C1 · p. 20
- The selected coefficients—α = 3 for Qwen and α = 4 for both Llama models—are a convention, not an optimum, because steering effects and coherence costs depend on direction.For Llama-8B in self-direction↔security at α = 4, CAA coherence is 91.0 under AaUa and 28.5 under AbUb.Source: §C.1 · p. 18
C.2 Alignment Stability
- At BASE, TV is highest for Qwen (0.89–1.05) and lowest for Llama-8B (0.40–0.48); SP and ASP mostly remain near unsteered levels.The ordering largely persists in steered conditions.Source: §C.2 · p. 18, Table C2 · p. 20
- CAA has the lowest total variation in some conditions, but where generation collapses its low scores reflect repetitive looping rather than stable value expression.For Llama-8B, CAA’s TV is 0.13–0.20 in four collapsed conditions, versus 0.39–0.77 for the other methods in those conditions.Source: §C.2 · p. 18, Table C2 · p. 20, Table C1 · p. 20
C.3 Repetitiveness
- CAA and SCAA show large repetitiveness increases in conditions where coherence collapses, while SP and ASP remain close to baseline levels, at most 0.20.For Qwen in self-direction↔security, CAA repetitiveness rises from 0.09 at BASE to 0.74 under AbUb and 0.69 under AbUa.Source: §C.3 · p. 21, Table C3 · p. 21
C.4 Trajectories with Standard Deviation
- The trajectory plots include standard-deviation bands, but the authors base conclusions on trajectory shapes rather than pointwise differences between conditions.The bands overlap for much of the conversation, so the main text shows averages for readability.Source: §C.4 · p. 21, Figure C2 · p. 21, Figure C3 · p. 21
D Examples of Conversations
- The unsteered Llama-70B example favors self-direction in eight of ten turns, with the lean receding when the user returns to concrete financial concerns.The excerpt is one run, not an average over seeds.Source: Table D1 · p. 22
- In the Llama-70B CAA example, steering first shifts the assistant from the user’s security framing toward independence, while the opposing AbUa example collapses into repetitive looping.The first excerpt remains coherent; the second is a single run in which steering opposes the model’s bias.Source: Table D2 · p. 23, Table D3 · p. 24