PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Multi-Value Control in LLMs via Causal Activation Steering

Payel Bhattacharjee, Ravi Tandon

arXiv:2609.30405v1cs.LG

TL;DR

Joint multi-value steering remains limited by fixed intervention strengths that cannot respond to a model’s evolving state. AIMES uses intermediate-layer vocabulary readouts to adapt value directions during decoding without training a separate state estimator. Its advantages over fixed joint and prompt steering depend on intervention depth, with smaller realized interventions than fixed steering and comparable response quality.

  • Problem

    Human-value steering has largely considered values in isolation, while fixed-strength joint steering cannot adapt to the model’s evolving internal state.

    Source: §1 · p. 2

  • Method

    AIMES constructs bipolar value directions and uses intermediate-layer vocabulary readouts as online observers to adapt multiple intervention strengths at each decoding step.

    Source: §1 · p. 2

  • Results

    Across five models, AIMES’s controllability advantages depend on objective and intervention depth, with favorable-depth gains over fixed and prompt steering, smaller realized interventions than fixed steering, and comparable response quality.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Within the Moral Foundations Theory values and model scales studied, online observer feedback provides lightweight, state-aware adaptation for single-pass multi-value steering.

    Source: §5 · p. 9

    The experiments cover five Moral Foundations Theory dimensions and synthetic and publicly available evaluation data, not a complete or universal representation of human values or direct human-subject experimentation.

    Source: ETHICS STATEMENT · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly deployed in settings where responses must reflect multiple, potentially interacting social norms and human values. Activation steering offers a lightweight alternative to training-based alignment by modifying internal activations at inference time. However, prior human-value steering methods have largely considered values in isolation, while direct composition of multiple directions relies on fixed intervention strengths that cannot respond to the model's evolving internal state. Motivated by this key observation, we introduce AIMES, a framework for adaptive multi-value activation steering. AIMES constructs layer-specific bipolar directions for moral-foundation values and uses intermediate-layer vocabulary readouts as online observers. An observer-guided controller then adapts the strength of each requested value intervention at every decoding step based on its current observed state, without training a separate value-state estimator. Across multiple instruction-tuned model families, value combinations, and intervention depths, we find that multi-value controllability varies across both value combinations and intervention locations. Compared with fixed joint steering and prompt-based steering, AIMES shows depth-dependent advantages that are broadly supported across two independent evaluators, with some variation in the precise depth at which specific control effects emerge. These advantages come with smaller realized activation-space interventions than fixed-joint steering and comparable response quality. Overall, our results suggest that online observer feedback can provide lightweight, state-aware adaptation for single-pass multi-value steering.

1 INTRODUCTION

Multi-value alignment must address interacting values, while fixed joint steering cannot adapt to the model’s changing internal state. AIMES uses vocabulary-readout feedback to adjust value interventions during decoding, and its evaluations examine how controllability varies by objective and depth.

1 INTRODUCTION

  • Existing preference-optimization approaches can require additional preference collection, optimization, or fine-tuning, making post-hoc adaptation to changing value requirements costly.Activation steering offers an inference-time alternative that modifies internal representations without updating model parameters.Source: §1 · p. 1
  • Prior human-value steering largely treats values individually, while jointly applying fixed-strength directions cannot adapt to changes in the model’s internal state during generation.Value directions may be correlated or conflicting, and intervention on one value can affect another.Source: §1 · p. 2
  • AIMES adapts the relative strength of multiple value-specific interventions at every decoding step using intermediate-layer observers, without training a separate value-state estimator.It frames joint value steering as closed-loop control rather than fixed-strength direction composition.Source: §1 · p. 2
  • The evaluation covers five models across three families and multiple multi-value objectives, testing adaptive control across intervention depths under two independent judges.The judges are GPT-5.6-Sol and Claude-Opus-4.8.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work establishes inference-time activation steering and explores both multi-attribute and multi-value control, but joint human-value activation steering remains comparatively underexplored. AIMES combines multi-value interventions with online feedback from intermediate vocabulary readouts.

Activation Steering for Human Values and Multiple Attributes

  • Human-value activation directions can alter foundation-relevant behavior, but cross-value effects and interference mean directions cannot generally be treated as independent.Multi-attribute methods address interference with gating, orthogonalization, subspace decomposition, or nonlinear direction construction, while interventions are typically fixed before generation.Source: §2 · p. 3

Steering Multiple Human Values

  • Multi-value approaches include prompting and combining separately generated outputs, while prior activation directions have been applied to values individually; joint activation-level control in one generation remains underexplored.AIMES is compared with fixed multi-value steering and prompt steering.Source: §2 · p. 3

Closed-Loop Activation Steering

  • Closed-loop steering adjusts intervention strength to evolving internal state, but typically relies on a separately trained classifier, probe, or fitted dynamical model.AIMES instead uses intermediate vocabulary readouts as observers while adapting multiple jointly applied value directions.Source: §2 · p. 3

Intermediate-Layer Vocabulary Readouts

  • Intermediate-layer readouts map hidden states to vocabulary-aligned coordinates, but visible representations do not necessarily imply causal controllability.The Logit Lens, Tuned Lens, and J-Lens use different readout mechanisms, and this visibility-controllability distinction matters when readouts provide feedback.Source: §2 · p. 4

3 AIMES: OBSERVER-GUIDED ADAPTIVE MULTI-VALUE STEERING

AIMES combines layer-specific value directions, intermediate-layer vocabulary observers, and a controller that adapts steering weights during generation. Its directions are bipolar, its observer is training-free, and its controller uses the current observed value state.

3 AIMES: OBSERVER-GUIDED ADAPTIVE MULTI-VALUE STEERING

  • AIMES has an offline direction-construction stage and an online steering stage, with the intervention layer fixed within each run and varied across experiments.Its three components are value directions, an intermediate-layer observer, and an observer-guided controller.Source: §3 · p. 4

3.1 VALUE-SPECIFIC STEERING DIRECTION ESTIMATION

  • Matched positive- and negative-pole examples produce normalized, layer-specific value directions that define bipolar axes for amplification or suppression.The Moral Foundations Theory contrasts include Care/Harm, Fairness/Cheating, Loyalty/Betrayal, Authority/Subversion, and Sanctity/Degradation; matched pairs reduce variation from topic, actors, setting, and writing style.Source: §3.1 · pp. 4–5
  • Care and Fairness are positively aligned, Loyalty and Authority are more weakly aligned and sometimes negatively aligned, and Sanctity aligns positively with Care and Fairness.These patterns motivate aligned, weakly coupled, and competing multi-value objectives, respectively.Source: Figure 2 · p. 4, §3.1 · p. 5

3.2 INTERMEDIATE-LAYER VALUE OBSERVER: DESIGN AND IMPLEMENTATION

  • AIMES estimates current value expression from intermediate-layer vocabulary readouts without training a separate value classifier; J-Lens is the default observer based on an ablation.The readout is fixed for a model and layer, shared across values and objectives, and uses fixed value-associated token sets independent of model activations.Source: §3.2 · p. 5

3.3 OBSERVER-GUIDED CAUSAL MULTI-VALUE STEERING

  • AIMES uses current observer scores to adapt each requested value direction’s contribution at every decoding step.The bounded value state is computed from standardized vocabulary evidence; amplification is weighted by 1 − c_k,ℓ,t and suppression by c_k,ℓ,t, and the controller is memoryless.Source: §3.2 · pp. 5–6, §3.3 · p. 6

4 EXPERIMENTAL ANALYSIS AND RESULTS

The experiments evaluate AIMES across five instruction-tuned models, three multi-value objectives, ten intervention depths, and two judges. Its advantages vary with objective and depth, with stronger cross-judge consistency for Care/Fairness retention than Sanctity suppression.

Experimental Setup and Baselines

  • The evaluation covers five instruction-tuned models, five Moral Foundations Theory dimensions, three joint objectives, and ten model-specific intervention depths.The baselines are fixed joint activation steering and prompt steering, assessed with two independent judges.Source: §4 · pp. 6–7

Overall Analysis of Adaptive Multi-Value Steering

  • Care/Fairness-retention advantages over fixed steering appear at about 30% depth across all five models, while Sanctity-suppression depth patterns shift between evaluators.Against prompt steering, Care/Fairness advantages are unanimous at several depths under GPT-5.6-Sol; the strongest Sanctity consensus is four of five models.Source: §4 · p. 7, Figure 3 · p. 7

Analyzing Prompt-Level Consistency

  • Prompt-level consistency is majority-positive for Care/Fairness against Prompt Steering across all five models at both displayed depths under GPT-5.6-Sol, but Claude’s results are more heterogeneous.Consistency against Fixed Multi-Value Steering varies more across models, depths, and evaluators; it measures how broadly mean advantages hold across prompts.Source: §4 · pp. 7–8, Table 1 · p. 8

Evaluating Balanced Multi-Value Control

  • AIMES beats Prompt Steering on GeoGain in all 25 plotted settings under both judges, while its advantage over fixed steering is model- and depth-dependent.GeoGain measures balanced movement in [∆C, ∆F, −∆S].Source: Figure 4 · p. 8, §4 · p. 9

Within-Response Steering Variation

  • At 30% depth, steering-coefficient ranges are nonzero across all five models and three objectives, spanning 0.45–0.83.The range measures within-response variation in steering strength; Figure 5 reports response means with 95% prompt-bootstrap confidence intervals.Source: Figure 5 · p. 8, §4 · p. 9

5 CONCLUSION

AIMES uses intermediate-layer vocabulary readouts to adapt activation-level steering of multiple values. In the studied models and Moral Foundations Theory values, its depth-dependent advantages come with smaller realized interventions than fixed joint steering and comparable response quality.

5 CONCLUSION

  • AIMES provides depth-dependent advantages over fixed joint steering and, at favorable depths, over prompt steering, with smaller realized interventions and comparable response quality.The study situates this lightweight approach within the Moral Foundations Theory values and model scales examined.Source: §5 · p. 9

AI USE STATEMENT

  • Generative AI assisted with vignette generation, code debugging, and manuscript editing; authors reviewed and verified the work and take full responsibility for it.AI-generated synthetic data underwent the validation and filtering procedures described in Appendix C.2.Source: AI USE STATEMENT · p. 9

ETHICS STATEMENT

  • The evaluation covers five Moral Foundations Theory dimensions and synthetic or public data, without direct human-subject experimentation; these dimensions are not a universal representation of human values.The paper notes that inference-time value control could also impose particular specifications or manipulate outputs undesirably.Source: ETHICS STATEMENT · p. 9

REPRODUCIBILITY STATEMENT

  • The paper reports that its main text covers methods, metrics, and setup, while the appendix documents generation procedures, intervention layers, ablations, geometry, and full layer-wise results.An anonymous code repository with implementation and supporting artifacts is linked in the experimental section.Source: REPRODUCIBILITY STATEMENT · p. 9

APPENDIX

  • The appendix is organized into background and preliminaries, the AIMES algorithm, detailed experimental setup, and additional experimental results.Source: APPENDIX · p. 14

A BACKGROUND AND PRELIMINARIES

Prior moral-value steering work establishes layer-specific directions but mainly intervenes on one value at a time, while jointly applied directions can interfere. AIMES addresses state dependence during generation using vocabulary-readout feedback to adapt intervention strengths.

Activation-Space Control and Direction Composition

  • Composing multiple activation directions can reduce control reliability, and many existing corrections fix the intervention rule before generation.Prior mechanisms include gating, orthogonalization, subspace decomposition, rotation, and nonlinear direction construction.Source: A BACKGROUND AND PRELIMINARIES · p. 14

Multi-Value Control

  • Multi-value control involves both geometric interference among directions and state dependence as the model's internal state evolves during generation.AIMES focuses on the latter by adapting jointly applied directions through online feedback.Source: A BACKGROUND AND PRELIMINARIES · p. 14

Feedback-Based Activation Steering

  • Earlier feedback-steering approaches typically obtain feedback from a trained classifier, probe, or fitted dynamical model, whereas AIMES uses an intermediate vocabulary readout directly.This avoids training a separate value-state estimator.Source: A BACKGROUND AND PRELIMINARIES · p. 14

Layer-Wise Vocabulary Readouts

  • J-Lens maps intermediate residual activations into vocabulary-aligned coordinates using a layer-specific average Jacobian to approximate downstream information propagation.Logit Lens directly applies the unembedding matrix, while Tuned Lens learns a layer-specific affine transformation.Source: A BACKGROUND AND PRELIMINARIES · pp. 14–15
  • Intermediate vocabulary readouts can support AIMES observer scores, but representational visibility does not necessarily coincide with steerability.Prior work reports that Logit-Lens accessibility can help predict steerability and depth, while effective directions may remain weakly exposed under that lens.Source: A BACKGROUND AND PRELIMINARIES · p. 15

Activation-Space Control and Direction Composition

  • Moral-foundation directions can shift foundation-relevant behavior, but prior work primarily studies values individually and finds cross-value effects.AIMES extends this contrastive-direction framework with matched positive-negative poles and adaptive joint control.Source: A BACKGROUND AND PRELIMINARIES · pp. 14, 16

B AIMES ALGORITHM

AIMES performs inference-time multi-value steering as a feedback loop: intermediate readouts estimate each value's current expression, and an observer-guided controller updates intervention weights during decoding.

B AIMES ALGORITHM

  • At each decoding step in the main experiments, AIMES estimates controlled values with an intermediate readout and recomputes their intervention coefficients.The intervention layer, readout, and base magnitude remain fixed within a response; the update interval is N=1, while larger N reduces readout cost.Source: B AIMES ALGORITHM · pp. 16–17
  • Each value's observer score is computed from a fixed set of ten surface-form tokens associated with its moral-foundation value.The token sets are fixed before evaluation and used consistently across models and steering objectives.Source: Table 2 · p. 16, B AIMES ALGORITHM · p. 16
  • AIMES applies observer-dependent weights to requested value directions jointly, supporting both single-value and multi-value objectives.Fixed Multi-Value Steering uses the same directions, layers, prompts, generation settings, and base magnitude but fixed coefficients; Prompt Steering expresses the objective in the input without activation modification.Source: B AIMES ALGORITHM · p. 16
  • The controller loops over generated tokens, reads the pre-intervention hidden state, computes value-specific scores and weights, then applies the joint adaptive intervention.Algorithm 1 describes this per-token sequence from input model and value directions to the generated response.Source: B AIMES ALGORITHM · p. 17

C.1 CONSTRUCTION AND OVERVIEW

The evaluation uses matched contrastive value directions, held-out prompts, and a depth sweep to study AIMES and its baselines. A J-Lens observer shows stronger directional responsiveness than Logit Lens across all five models, while downstream advantages vary by metric and architecture.

C.1 CONSTRUCTION AND OVERVIEW

  • Each foundation's steering direction is derived from 200 matched positive-negative vignette pairs, with evaluation prompts kept separate from direction-construction examples.Fixed Multi-Value Steering and AIMES use the same precomputed directions, so their comparison isolates intervention strategy rather than direction estimates.Source: §C.1 · p. 17
  • The main evaluation uses held-out MFRC prompts filtered so their source foundation is not among the directly controlled values.The two-value objectives retain 60 prompts each, the three-value objective retains 40, and all are evaluated at ten intervention depths against AIMES, fixed steering, and prompt steering.Source: §C.1 · p. 17

C.2 CONTRASTIVE VIGNETTE GENERATION

  • All five foundation-specific vignette sets show positive semantic contrasts against Social Norm under all three embedding models, with mean differences from 0.035 for Authority to 0.078 for Sanctity.The authors treat this as a corpus-level sanity check; the primary validation of steering directions is in target-model activation space.Source: §C.2 · pp. 18–19, Table 4 · p. 19
  • The contrastive corpus pairs positive and negative vignettes that preserve setting and structure while reversing the target moral orientation.Across five foundations, the corpus contains 1,000 pairs; twelve contexts provide diversity controls, and pairwise activation differences yield the same direction as the matched-set mean difference.Source: §C.2 · pp. 19–20

C.3 OBSERVER ABLATION: J-LENS VS. LOGIT LENS

  • J-Lens yields higher directional responsiveness than Logit Lens in all five models, while downstream AIMES-JL advantages are not uniform across architectures.AIMES-JL has higher worst-target gain in four of five models, lower non-target interference in three of five, and higher response quality in four of five; most behavioral confidence intervals overlap zero.Source: Figure 7 · p. 21, §C.3 · p. 21

C.4 LAYER-WISE GEOMETRY OF VALUE DIRECTIONS

Value-direction geometry changes across models and layers, and AIMES’s balanced-control advantage depends on intervention depth. Across the tested settings, AIMES also uses smaller realized activation updates, while its coefficient adaptation and response quality are assessed separately.

C.4 LAYER-WISE GEOMETRY OF VALUE DIRECTIONS

  • Value-direction geometry varies by model and depth: Qwen3 directions become more positively correlated from early to later layers, while Llama-3.1-8B-Instruct similarities are mostly positive and relatively stable.Loyalty is less aligned with other foundations in Llama, and is nearly orthogonal or weakly anti-aligned with some directions in early Qwen3 layers.Source: §C.4 · p. 21, Figure 8 · p. 22
  • Gemma models show especially large depth-related geometry changes, including early negative pairings and more positive middle- or late-layer correlations.These shifts mean simultaneous value interventions can interact differently by intervention depth.Source: §C.4 · p. 21
  • AIMES reduces realized activation perturbations by approximately 8%–87% across 150 model-layer-objective settings, with confidence intervals excluding zero throughout.Reduction magnitude varies by model, layer, and objective rather than reflecting constant rescaling.Source: §C.4 · p. 21, Figure 9 · p. 23
  • Across intervention depths, AIMES’s balanced-control advantage varies substantially against Fixed Multi-Value Steering but is more consistently positive against Prompt Steering.The depth dependence is especially pronounced against fixed joint steering for the competing three-value objective.Source: §C.4 · p. 23, Figure 10 · p. 23

D ADDITIONAL EXPERIMENTAL RESULTS

  • GeoGain measures balanced progress across a composite objective, rewarding movement in every requested direction and penalizing imbalanced control.Comparisons use prompt-paired differences before averaging within each model-layer setting.Source: D ADDITIONAL EXPERIMENTAL RESULTS · p. 24
  • The coefficient-range diagnostic measures within-response steering variation, but variation alone does not establish behavioral benefit; response quality is separately scored for coherence, fluency, and relevance.Quality changes are reported relative to the corresponding unsteered response, with comparisons computed at the exact intervention layer.Source: D ADDITIONAL EXPERIMENTAL RESULTS · p. 24

D.1 ROBUSTNESS ANALYSIS

Depth-wise prompt-level consistency is reported for GPT-5.6-Sol and Claude-Opus-4.8. Claude’s consistency varies across models, objectives, and depths, with positive-majority consistency across a broad range of settings.

D.1 ROBUSTNESS ANALYSIS

  • The GPT-5.6-Sol robustness results are presented as full depth-wise prompt-level consistency.The supplied passage gives the analysis scope but no outcome values.Source: §D.1 · p. 24
  • Table 5 reports tie-adjusted AIMES consistency percentages with pointwise 95% prompt-bootstrap confidence intervals over 40 matched prompts per cell.The supplied passage identifies the table’s measurement and sample size but not its cell values.Source: Table 5 · p. 25
  • Under Claude-Opus-4.8, positive-majority prompt-level consistency appears across a broad range of settings for both comparison methods, though consistency varies by model, objective, and depth.Values above 50% indicate that AIMES’s advantage occurs for a majority of matched prompts.Source: §D.1 · p. 26, Table 6 · p. 26

D.2 DETAILED RESPONSE-QUALITY RESULTS

A blind judge’s comparisons indicate broadly comparable overall response quality for AIMES, while relevance more often favors AIMES over Prompt Steering.

D.2 DETAILED RESPONSE-QUALITY RESULTS

  • The response-quality counts are descriptive rather than significance tests, and the blind judge sees only the original prompt and generated response.The steering method, intervention layer, objective, and source foundation are hidden from the judge.Source: Table 7 · p. 27, §D.2 · p. 27
  • AIMES has higher mean overall quality than Fixed Multi-Value Steering in 76/150 settings and Prompt Steering in 81/150 settings.These counts indicate broadly comparable quality rather than a consistent quality advantage.Source: §D.2 · p. 27, Table 7 · p. 27
  • AIMES exceeds Prompt Steering on relevance in 106/150 settings (70.7%).The judge scores coherence, fluency, and relevance on a four-point scale.Source: §D.2 · p. 27

D.3 LAYER-WISE WITHIN-RESPONSE STEERING VARIATION

AIMES’s steering coefficients vary within responses across models, objectives, and intervention depths. This variation persists across the sampled depth grid rather than being confined to the representative 30% depth.

D.3 LAYER-WISE WITHIN-RESPONSE STEERING VARIATION

  • The diagnostic averages each target-value coefficient’s within-response range across controlled values and target-disjoint prompts, with larger ranges indicating greater steering-strength variation.It measures coefficient variation only and does not by itself establish behavioral benefit.Source: §D.3 · pp. 27–28
  • Across all five models, coefficient ranges remain nonzero at all ten sampled depths, showing within-generation steering-strength changes across the network.The reported objectives include two-value combinations and the competing three-value objective.Source: §D.3 · p. 28, Figure 11 · p. 28
  • Mean coefficient ranges span approximately 0.45–0.84 across the plotted conditions, with variation across early, middle, and later layers in all five models.The pattern occurs for both two-value objectives and the competing three-value objective.Source: §D.3 · p. 28
  • The representative 30% depth is not isolated: adaptive coefficients remain responsive across the complete sampled depth set.Layer-wise results describe meaningful within-response changes rather than approximately fixed steering weights.Source: §D.3 · p. 28

D.4 INDEPENDENT-JUDGE-BASED ANALYSIS OF DEPTH-WISE AND GEOGAIN RESULTS

Under Claude-Opus-4.8, AIMES retains a balanced-control advantage for the three-value objective, while the precise depth of some advantages varies by evaluator. The 30% Care/Fairness-retention result is the most evaluator-robust depth-specific finding reported here.

Depth-wise interaction contrasts

  • At 30% depth, the Care/Fairness-retention advantage over Fixed Multi-Value Steering is positive for all five models under both judges.The Prompt Steering comparison is also positive for all five models at 30% depth, but some other depths unanimous under GPT-5.6-Sol are not unanimous under Claude.Source: §D.4 · pp. 28–29, Figure 12 · p. 29

GeoGain Analysis

  • For (↑Care ↑Fairness ↓Sanctity), AIMES beats Prompt Steering in 50/50 and Fixed Multi-Value Steering in 30/50 model–depth settings.The 50/50 result strengthens the 47/50 count under GPT-5.6-Sol; against Fixed Multi-Value Steering, unfavorable cells are near ties while favorable cells reach approximately +0.065 GeoGain.Source: Figure 13 · p. 29, §D.4 · p. 30

Depth-wise interaction contrasts

  • The depth of the Sanctity-suppression advantage over Fixed Multi-Value Steering is evaluator-sensitive, with strongest consensus near 60% under GPT-5.6-Sol and near 10% under Claude.Against Prompt Steering, the comparison remains broadly positive at several depths but is not unanimous at any depth under either judge.Source: §D.4 · pp. 28–29

GeoGain Analysis

  • Claude’s GeoGain comparison with Prompt Steering depends on the objective: AIMES is lower in 50/50 Care/Fairness settings, higher in 21/50 Loyalty/Authority settings, and higher in 50/50 three-value settings.On the three-value objective, Gemma-3-12B shows an approximately +0.06 to +0.085 advantage across depth despite being disadvantaged on the simpler two-value objectives.Source: §D.4 · p. 30

Related papers

Loading 2609.30405v1…