PaperlayerView exact revision on arXiv

Source-linked AI summary

Spillover-Aware Multi-Value Steering for Pluralistic LLM Alignment

Weici Pan, Xander Barron, Jiawei Zhou, Zhenhua Liu

arXiv:2609.05800v1cs.AI

TL;DR

Existing activation steering lacks reliable multi-dimensional control because steering one value can spill into others, a gap that matters for pluralistic alignment. The paper uses Gram-matrix geometry to correct multi-value steering without retraining or auxiliary models. On climate discourse, NSE rises from +5.9% to +14.0% over 100,000 pairwise judgments.

  • Problem

    Existing activation steering handles one concept at a time, while pluralistic alignment requires different stakeholders’ value emphases to be controlled simultaneously without spillover.

  • Method

    The paper discovers value dimensions, extracts steering directions, diagnoses their Gram-matrix entanglement, and applies G−1 correction in an automated pipeline without fine-tuning, reward models, or manual prompt engineering.

  • Results

    14.0% NSE versus 5.9% for uncorrected methods was achieved on climate discourse, validated over 100,000 pairwise judgments.

  • Takeaways & Limitations

    Inverse-Gram correction restores dimension-specific control for arbitrary preference vectors and steering strengths within the evaluated setting.

  • Takeaways & Limitations

    Validation covers climate discourse and a replication on Medical Ethics, while broader multi-domain validation remains future work; the full optimal correction is numerically unstable in the current additive regime.

Abstract

from arXiv · show

Activation steering controls LLM behavior at inference time by adding learned directions to hidden states, but existing methods handle one concept at a time. Pluralistic alignment, where different stakeholders need different value emphases, requires steering multiple dimensions simultaneously. We show that naive steering produces substantial spillover: the effect intended for one value leaks into others. This parallels the treatment-versus-spillover decomposition in causal inference. We trace spillover to geometric entanglement of steering directions, captured by their Gram matrix, and derive a zero-cost correction from an activation-norm-penalized objective that decouples each direction's contribution exactly. Our end-to-end pipeline requires no fine-tuning, no reward model, and no manual prompt engineering: given only domain questions, it automatically discovers value dimensions, extracts directions, diagnoses entanglement, and applies corrected steering. On climate discourse, the correction improves the net steering effect from +5.9% to +14.0%, validated over 100,000 pairwise judgments.

1 Introduction

Pluralistic alignment requires independently controllable value emphases, but existing activation steering handles one concept at a time and naive multi-direction steering causes spillover. The paper formalizes this failure with NSE and introduces Gram-matrix correction alongside an automated end-to-end pipeline.

  • Different stakeholders require different value emphases, whereas standard alignment produces responses reflecting averaged preferences.
  • Naively combining steering directions causes effects intended for one value to leak into others, undermining dimension-specific control.
  • NSE separates targeted impact from non-target spillover, with NSE ≈0 indicating uniform rather than dimension-specific steering.
  • 5.9% is the best uncorrected NSE, and increasing steering strength does not improve specificity because target and leakage scale together with the Gram matrix.
  • The pipeline discovers value dimensions, extracts steering directions, diagnoses geometric entanglement, and applies corrected multi-value steering without fine-tuning, reward models, or manual prompt engineering.
  • 14.0% NSE versus 5.9% for uncorrected methods was achieved on climate discourse with 100,000+ pairwise judgments.

2 Related work

Prior pluralistic-alignment methods either require retraining or offer limited continuous control in prompt space, while activation steering has largely remained single-concept. Multi-property methods identify interference but rely on heuristics or training without broad guarantees.

  • Training-time pluralistic-alignment methods require retraining when stakeholder preferences change.
  • Prompt-based inference-time methods avoid retraining but make fine-grained continuous value control difficult to express.
  • Existing activation-steering methods handle one concept at a time and do not evaluate unintended effects on non-target behaviors.
  • Prior multi-property steering addresses feature entanglement heuristically or through training, with limited scalability and no guarantee.

3 Preliminaries

The method represents simultaneous steering as a linear combination of learned directions mapped from stakeholder preference vectors. It measures specificity with NSE, contrasting target emphasis against average non-target spillover.

  • Activation steering adds learned direction vectors to hidden states, using h′ = h + αd for a single concept.
  • For K dimensions, D collects steering directions and α contains the simultaneous steering coefficients.
  • A preference vector w specifies requested value emphases, and a linear correction matrix H maps preferences to steering coefficients.
  • The Gram matrix G = D⊤D captures pairwise direction geometry and quantifies entanglement; larger condition numbers indicate greater entanglement.
  • Under naive steering, projection onto a non-target direction equals its Gram-matrix inner product with the target, producing geometric leakage.
  • NSE equals target effect minus spillover, where target effect measures increased target emphasis and spillover averages increased non-target emphasis.
  • NSE = 0 indicates uniform emphasis without dimension-specific control, while positive NSE indicates preferential targeting.

4 Method

The method diagnoses direction entanglement before generation and corrects it by applying the inverse Gram matrix. Its dual-track pipeline uses probes to validate value dimensions and contrast directions to steer them.

  • Probe directions are near-orthogonal, with κ(G) = 1.75 and off-diagonal entries below 0.20.
  • Probe and contrast directions occupy near-orthogonal subspaces, separating value detection from value steering.
  • 4.1 Geometric correction: Setting H = G−1 makes each direction’s contribution equal its requested weight and removes cross-dimensional leakage geometrically.
  • 4.1 Geometric correction: The correction requires only replacing w with G−1w before forming the steering vector, with no additional training, reward model, generation, or evaluation.
  • 4.2 Pipeline: The pipeline uses probes to discover and validate value dimensions, while contrast-based directions provide the actual steering vectors.
  • 4.2 Pipeline: It computes G from contrast directions, inspects κ(G), applies G−1, and forms Dα for each stakeholder preference.

5 Experiments

Experiments show that inverse-Gram correction consistently improves dimension-specific steering across direction types, model scales, architectures, and numbers of jointly steered values. The strongest gains occur where geometric entanglement makes naive steering fail, while direction types retain complementary strengths across target dimensions.

  • 5.2 Main results: 14.0% NSE versus 5.9% for uncorrected contrast steering on Llama-3.2-3B-Instruct, with no uncorrected method exceeding 6%.The difference is statistically significant (z = 3.40, p < 0.001).
  • 5.2 Main results: Contrast directions gained +8.1 percentage points from correction, compared with +4.1 pp for RepE and +3.2 pp for ActAdd.The correction improved every generative direction type, while layer separation reached only 1.5%.
  • 5.2 Main results: Contrast_ginv suppressed spillover to 42.9%, whereas RepE_ginv left spillover near chance at 49.5% while increasing target emphasis.Contrast directions concentrated 70.8% of variance in one shared eigenvector, which inverse-Gram correction removes.
  • 5.2 Main results: As jointly steered values increased, naive macro NSE fell below chance at K = 8 and K = 10, while corrected NSE remained positive through K = 12.The Gram condition number rose from 24.4 at K = 4 to 329.7 at K = 12; corrected NSE was +9.0%, +6.7%, and +17.8% at K = 8, 10, and 12.
  • 5.2 Main results: Per-dimension performance was complementary: contrast_ginv peaked on SocialJustice at +23.2%, while RepE_ginv peaked on FutureEthics at +11.2%.RepE_ginv was negative on SocialJustice at −5.4%, showing that correction cannot create a missing dimension signal.
  • 5.3 Cross-scale validation: Corrected steering improved every tested direction type on Llama-3.1-8B-Instruct and improved contrast_ginv and actadd_ginv on Qwen2.5-7B-Instruct.On 8B, actadd_ginv reached 6.0% versus 1.6% naive; on Qwen, contrast_ginv gained +4.7 pp and actadd_ginv +2.5 pp.
  • 5.3 Cross-scale validation: Direction-type rankings varied by model scale, but the corrected variant consistently outperformed its naive counterpart.The strongest corrected method was contrast on 3B and ActAdd on 8B; differences among corrected 8B methods were within the noise margin.
  • 5.4 Discussion: Geometric decorrelation removes deterministic, input-independent leakage before nonlinear generation, while residual interference can arise outside the probe subspace.The correction is exact in direction geometry, and off-diagonal Gram entries of 0.7–0.85 indicate substantial pre-generation leakage.

6 Conclusion

The paper concludes that inverse-Gram correction restores dimension-specific control in multi-value activation steering without additional training or generation cost. It raises NSE on climate discourse and replicates gains across models and architectures, while broader validation and a stable full optimal correction remain open.

  • 6 Conclusion: Inverse-Gram correction eliminates cross-dimensional leakage by construction for any preference vector and steering strength.The method formalizes spillover with NSE and requires no additional training, reward model, or generation cost.
  • 6 Conclusion: 14.0% NSE versus +5.9% for uncorrected steering on climate discourse was validated over 100,000 pairwise judgments.The gains were consistent across Llama-3.1-8B and Qwen2.5-7B.
  • 6 Conclusion: The pipeline was replicated from scratch on Medical Ethics, but broader multi-domain validation remains future work.The conclusion limits the current empirical scope to climate discourse and one additional domain.
  • 6 Conclusion: The full optimal correction H* is numerically unstable in the current additive steering regime, and a unified direction type remains open.The unresolved direction type would need to be simultaneously discriminative, steerable, and geometrically clean.

A Value taxonomy construction details

The paper compresses 12 candidate value dimensions into four data-driven dimensions, then validates their interpretability, stability, and probe-based detectability across datasets and models.

  • Compression to four dimensions: 12 candidate dimensions are grouped into Accuracy, TechEcon, SocialJustice, and FutureEthics using inter-dimension correlations.Accuracy remains standalone; the other dimensions form present-oriented justice, future-oriented ecological, and technology-economics clusters.
  • Compression to four dimensions: The four compressed dimensions occur at rates of 91.4% Accuracy, 38.2% TechEcon, 27.6% SocialJustice, and 49.3% FutureEthics.
  • Interpretability: The taxonomy aligns post hoc with established moral-psychology and environmental-ethics frameworks without being designed from them.Accuracy maps to competence-oriented values, SocialJustice to universalism-benevolence, FutureEthics to intergenerational justice and precaution, and TechEcon to pragmatic cost-benefit reasoning.
  • Stability: Removing one sub-dimension changes at most 14.4% of compressed labels, while SocialJustice changes no more than 3.3%.The grouped dimensions retain information that dominant members alone would miss.
  • Question types: Clustering identifies four interpretable question types: PureAccuracy 43.6%, FutureEthics-focused 17.6%, PolicyComplex 21.8%, and TechEcon-focused 17.1%.PolicyComplex questions engage all four value dimensions, whereas PureAccuracy questions have minimal value loading.
  • Probe validation: Logistic-regression probes detect the dimensions across domains and models, with Accuracy reaching F1 0.93 and cross-domain transfer losing only 0.04 mean F1.Probe geometry also becomes more conditioned with scale: κ(Gprobe) decreases from ∼10 at 1B to ∼2 at 8B.
  • Geometric validation: Generative directions are substantially more entangled than discriminative probe directions, with generative κ(G) ≥7.8 versus probe κ(G) ≤1.9.Generative Gram matrices also concentrate 55–77% of variance in one eigenvalue, unlike probe matrices.

D Detection and control occupy different subspaces

Probe directions detect value information, whereas contrast-based directions produce value shifts; the two occupy near-orthogonal subspaces and require separate roles in the pipeline.

  • Detection versus control: Probe directions have mean F1 0.78 yet share near-zero cosine similarity with contrast directions that produce measurable steering effects.This indicates that detection and control are not merely different magnitudes of the same representation.
  • Near-orthogonality: All 12 matched probe–contrast cosines fall between 0.01 and 0.09, not significantly above the random baseline in R4096.The observed values are not significantly above random (p = 0.89), and cross-pair cosines remain below 0.06.
  • Layer stability: The probe–contrast separation remains stable across layers 8–28, with mean absolute cosine 0.025–0.037 and κ(Gcontrast) ranging from 13 to 24.Probe condition numbers remain within 1.74–1.88 across the same layer range.
  • Causal specificity: Contrast-based personas produce positive target lifts averaging 2.20 points, while SocialJustice also lifts FutureEthics by 0.90 points.The off-diagonal effect is attributed to thematic overlap between environmental justice and intergenerational ethics.
  • Causal specificity: The lift matrix measures text-level persona prompting rather than activation steering, establishing that the generative shift captured by contrast directions is dimension-specific.
  • Pipeline implication: The dual-track pipeline uses probes to discover and validate the taxonomy and contrast directions to perform steering corrected for entanglement.A unified direction type that is both discriminative and steerable remains an open question.

E Dead-end experiments

Several alternatives fail because they do not produce reliable positive value-specific effects, while causal-Jacobian and Fisher-weighted corrections are too noisy or ill-conditioned at useful intervention scales.

  • Conceptor steering: Conceptor steering tested across 211 configurations achieves a best mean diagonal lift of −0.17, steering in the wrong direction.Simple additive probe steering is also negative at −0.14 in the comparison.
  • Contrastive decoding: Contrastive decoding degenerates into repetitive fragments with all value scores at zero across uniform and per-dimension configurations.The method conflates value-relevant tokens with coherent tokens, which is inadequate for nuanced multi-value emphasis.
  • Layer assignment: Layer-specific direction assignment does not improve outcomes: all 24 layer permutations vary by less than 1.0 point on every dimension.The heuristic therefore does not address the underlying geometric coupling in this setting.
  • Causal-Jacobian estimation: The estimated causal Jacobian has average absolute diagonal entries of ∼0.01, off-diagonals comparable to diagonals, and no dominant diagonal structure.Per-question causal effects are highly variable, with coefficient of variation 11.8, and correlate with question type.
  • Fisher weighting: The Fisher-weighted Gram matrix has entries near 10^-4 versus roughly 1 for G, so its inverse amplifies estimation noise.Both gradient-based and KL-based estimates produce the same order of magnitude.
  • Scope boundary: At useful perturbation magnitudes near 15%, the first-order approximation is valid only below 1%, limiting the causal-Jacobian correction.The G−1 component remains the computable and useful geometric correction in the current additive-steering regime.
  • Full correction: The full H* solution yields ∥α*∥≈3.4 × 10^6 and, after rescaling, produces near-uniform coefficients that do not outperform G−1.Five-question pairwise tests confirm that the M-dependent component adds noise rather than signal.

H Evaluation methodology

The evaluation compares steered responses with matched unsteered baselines using repeated pairwise judgments across all value dimensions, with checks for resolution, agreement, scale, and length confounds.

  • Pairwise protocol: Each method–target–question comparison generates a steered response and an identically parameterized unsteered baseline, judged independently on all K = 4 value dimensions.
  • Pairwise protocol: Each judgment is repeated three times with randomized response order, and majority voting resolves the final verdict.Ties count as 0.5 for both sides.
  • Resolution: Pairwise comparison detects 55–65% win rates where absolute scoring gives R2 < 0.03 between steering strength and value scores.The protocol targets subtle shifts in value emphasis that coarse absolute scoring misses.
  • Judge agreement: Across 86,400 individual 3B judgments, calls agree unanimously on 62%; the final distribution is 25.0% steered wins, 25.5% baseline wins, and 49.5% ties.
  • Confound checks: Response-length differences correlate with verdicts at r < 0.28, and restricting comparisons to responses within 20% of equal length leaves the NSE ranking unchanged.
  • Evaluation scale: The reported NSE evaluations contain 350,910 judge votes over 128,270 resolved pairwise comparisons.The main experiments account for 182,400 votes across the 3B, 8B, and Qwen runs.

I Steering strength sensitivity

Across steering strengths, G−1 correction consistently improves NSE over naive steering, while the uncorrected gap widens as perturbations increase. The operating points used in the main experiments are supported by strength sweeps, but small sweep samples make exact NSE values indicative.

  • 8B sweep: At α = 20, ε = 0 is the best 8B configuration, supporting pure G−1 correction over regularized variants.The 4 × 4 Gram matrix has κ = 11.45, so damping removes signal rather than noise in this setting.
  • Strength sweeps: G−1 correction improves NSE at every tested alpha level, with the advantage widening as steering strength increases.The 3B contrast sweep reports a gap from +7.3 pp at α = 15 to +21.0 pp at α = 45.
  • Mechanism: Naive steering amplifies target and spillover together as α rises, whereas G−1 raises target emphasis while suppressing spillover below chance.This follows the geometric interpretation that increasing α amplifies the shared component and G−1 removes it.
  • Robustness: G−1 remains effective across a factor of 3 in steering strength and avoids catastrophic collapse at high α.The 3B contrast sweep reaches NSE = +32.7% at α = 45, although the collapse threshold may depend on direction type and model scale.
  • Caveat: Sweep estimates use only n = 20 questions per configuration, so precise NSE values are indicative despite robust directional trends.The main-text comparisons use larger samples: n = 200 for 3B and n = 100 for 8B.

J.1 3B per-dimension results (all 9 methods)

Per-dimension results show that corrected generative directions provide positive specificity across dimensions, but their profiles vary by direction type and model scale. Probe correction adds little because probe directions are already near-orthogonal and weakly steerable.

  • 3B per-dimension results: Contrast_ginv achieves positive NSE on all four dimensions, strongest on SocialJustice (+21.7%) and weakest on FutureEthics (+4.6%).
  • 3B per-dimension results: RepE_ginv is strongest on TechEcon (+12.2%) and FutureEthics (+9.4%) but fails on SocialJustice (−5.1%).The complementary profiles confirm that correction performance depends on the direction type and target dimension.
  • 3B per-dimension results: Probe_naive and probe_ginv perform similarly, with mean NSE of +5.6% and +4.3%, respectively.The near-orthogonal probe geometry has κ = 1.90, leaving little for G−1 correction to improve.
  • 8B per-dimension results: On 8B, actadd_ginv is the most consistently positive direction type, while contrast_ginv is negative on Accuracy (−0.8%) but positive on the other dimensions.The best direction type reverses relative to 3B, yet G−1 improves every generative direction type on both scales.
  • Value discovery: Unsupervised clustering across 10 domains recovers question format and topic rather than value structure, motivating supervised annotation.All domains select K = 2 with low silhouette scores, and cluster centroids across domains have cosine similarity above 0.90.

L Extended discussion

The extended discussion frames G−1 correction as a geometric remedy for leakage in steerable but entangled directions, while distinguishing that remedy from semantic value relationships. Its scope is limited by input dependence, modest effects, nonlinear residuals, judge bias, and incomplete domain validation.

  • Interpretation: Causal-inference framing separates target steering from unintended spillover, while G−1 redistributes steering energy without additional tuning cost.The treatment is steering toward a target value, the direct effect is increased target emphasis, and interference is non-target emphasis change.
  • Entanglement and steerability: Generative directions are steerable but entangled, and greater entanglement corresponds to larger G−1 gains.On 3B, contrast has κ = 24.4 and gains +8.1 pp, whereas ActAdd has κ = 14.2 and gains +3.2 pp.
  • Deployment diagnostic: The Gram matrix provides a lightweight pre-deployment audit: high κ(G) signals poor specificity without correction, and off-diagonal entries identify interfering value pairs.The diagnostic requires probe training but no generation.
  • Limitations: The pipeline remains limited by domain coverage, input-dependent effects, modest target magnitude, nonlinear residual interference, and LLM-judge evaluation.The authors identify climate as the main end-to-end validation domain, while broader validation and human evaluation remain future work.
  • General preference vectors: For any preference vector w, α = G−1w makes the perturbation’s projection onto each steering direction equal to its requested weight.The guarantee follows algebraically from GG−1w = w and extends beyond one-hot preferences.

O Cross-architecture validation: Qwen2.5-7B-Instruct

Qwen2.5-7B-Instruct reproduces the main geometric pattern and benefits from G−1 correction, extending the result beyond the Llama family. However, FutureEthics remains weak across methods, showing that correction cannot create missing value-specific signal.

  • Aggregate results: On Qwen2.5-7B-Instruct, contrast_ginv reaches +7.2% NSE versus +2.5% for contrast_naive, a +4.7 pp improvement.ActAdd correction also improves over its naive counterpart by +2.5 pp.
  • Cross-model patterns: Qwen shows higher target effects but also higher spillover than Llama models, with target effects of 60–65% versus 52–57%.The corresponding Qwen spillover is 57–58%, compared with 48–50% for Llama models.
  • Per-dimension results: FutureEthics is the weakest Qwen dimension for every method, with negative NSE for both corrected and uncorrected variants.The result indicates that the Qwen FutureEthics contrast direction lacks sufficient value-specific signal in this domain.
  • Entanglement structure: Qwen reproduces the two-camp geometry: probe directions are near-orthogonal, while generative directions are entangled.Generative condition numbers are κ = 10.7–11.8, and the SJ–FE off-diagonal is largest in both generative matrices.
  • K-scaling: The correction’s gain over naive steering remains positive and significant as the number of value dimensions scales beyond K = 4.Table 24 uses the main-text K = 4 sample size for comparison and smaller samples for K ≥ 8.

P Scaling the number of value dimensions

As the number of jointly steered value dimensions increases, geometric entanglement worsens, while inverse-Gram correction maintains positive macro gains and recovers targets harmed by naive steering. These benefits extend across domains, judges, and layer choices, though individual judgments remain moderately reliable.

  • Geometric scaling: κ(G) rises from 24.4 at K = 4 to 329.7 at K = 12, indicating increasingly ill-conditioned joint steering.The condition number increases at K = 8, 10, and 12 as more directions are steered simultaneously.
  • Geometric scaling: +12.5pp at K = 8, +10.1pp at K = 10, and +11.9pp at K = 12 are inverse-Gram macro gains over naive steering.Each gain is statistically significant under both Wilcoxon signed-rank and paired bootstrap tests, and each is at least as large as the +8.1pp gain at K = 4.
  • Per-target recovery: +29.7pp recovery brings Accuracy from −33.4% to −3.7% at K = 8, while other entangled targets show comparable recoveries.At K = 10, Accuracy improves from −29.6% to −0.6%; Equity_Present improves by +30.0pp at K = 8, and Efficiency by +27.1pp at K = 10.
  • Conditioning and compression: +6.0% naive to +17.8% corrected at K = 12 shows that the macro gain survives the uncompressed taxonomy.Compression improves conditioning, but the correction remains effective without compression.
  • Cross-domain scaling: +14.2% corrected NSE versus +2.5% naive in medical ethics yields a +11.7pp gain, showing usefulness beyond the more entangled climate setting.The gain is largest for Evidence (+35.3pp), whose direction is most orthogonal to the normative dimensions.
Loading 2609.05800v1…