PaperlayerView exact revision on arXiv

Source-linked AI summary

YaPO: Learnable Sparse Activation Steering Vectors for Domain Adaptation

Abdelaziz Bounhar, Rania Hossam Elmohamady Elbadry, Hadi Abdine, Preslav Nakov, Michalis Vazirgiannis, Guokan Shang

arXiv:2601.08441v1cs.AI

TL;DR

Dense steering vectors can entangle latent factors, limiting stable and fine-grained alignment. YaPO addresses this by learning reference-free sparse steering vectors in a pretrained SAE space, and the paper reports faster convergence, greater stability, stronger cultural alignment, and broader alignment generalization.

  • Problem

    Dense steering vectors entangle multiple latent factors because of neuron multi-semanticity, limiting stability and effectiveness in fine-grained alignment settings.

  • Method

    YaPO learns trainable sparse steering vectors directly in a pretrained SAE latent space using a reference-free BiPO-style preference objective.

  • Results

    YaPO converges faster, improves stability and interpretability, outperforms dense and baseline models on cultural alignment, and generalizes to hallucination, wealth-seeking, jailbreak, and power-seeking.

  • Takeaways & Limitations

    Sparse-space preference optimization provides a general recipe for efficient, stable, and fine-grained alignment across cultural and other alignment dimensions.

  • Takeaways & Limitations

    Experiments use only Gemma-2 2B and 9B models, and the cultural dataset captures cross-country but not within-country diversity.

Abstract

from arXiv · show

Steering Large Language Models (LLMs) through activation interventions has emerged as a lightweight alternative to fine-tuning for alignment and personalization. Recent work on Bi-directional Preference Optimization (BiPO) shows that dense steering vectors can be learned directly from preference data in a Direct Preference Optimization (DPO) fashion, enabling control over truthfulness, hallucinations, and safety behaviors. However, dense steering vectors often entangle multiple latent factors due to neuron multi-semanticity, limiting their effectiveness and stability in fine-grained settings such as cultural alignment, where closely related values and behaviors (e.g., among Middle Eastern cultures) must be distinguished. In this paper, we propose Yet another Policy Optimization (YaPO), a \textit{reference-free} method that learns \textit{sparse steering vectors} in the latent space of a Sparse Autoencoder (SAE). By optimizing sparse codes, YaPO produces disentangled, interpretable, and efficient steering directions. Empirically, we show that YaPO converges faster, achieves stronger performance, and exhibits improved training stability compared to dense steering baselines. Beyond cultural alignment, YaPO generalizes to a range of alignment-related behaviors, including hallucination, wealth-seeking, jailbreak, and power-seeking. Importantly, YaPO preserves general knowledge, with no measurable degradation on MMLU. Overall, our results show that YaPO provides a general recipe for efficient, stable, and fine-grained alignment of LLMs, with broad applications to controllability and domain adaptation. The associated code and data are publicly available\footnote{https://github.com/MBZUAI-Paris/YaPO}.

1 Introduction

YaPO addresses the challenge of fine-grained, interpretable behavior control by learning sparse steering vectors in a pretrained SAE latent space. The method targets cultural alignment and broader alignment behaviors while improving convergence, stability, and interpretability over dense approaches.

  • Motivation: Fine-grained and interpretable control remains difficult because RLHF is costly and inflexible, while prompt engineering is brittle and less efficient than fine-tuning.These approaches also provide limited transparency into how specific behaviors are modulated.
  • Motivation: Dense steering vectors entangle multiple latent factors through neuron multi-semanticity and superposition, limiting stability, interpretability, and fine-grained alignment.SAS improves interpretability with SAE features but uses static averaged activations rather than learnable sparse vectors.
  • Method: YaPO is a reference-free method that learns trainable sparse steering vectors in a pretrained SAE latent space using a BiPO-style objective.It combines preference optimization with sparse-space interpretability and minimal training overhead.
  • Evaluation: YaPO’s cultural benchmark spans five language families and fifteen cultural contexts, including fine-grained distinctions among same-language cultures.The study reports an implicit–explicit localization gap in baseline models and evaluates generalization on MMLU and established alignment benchmarks.
  • Results: YaPO converges faster, improves training stability, and yields more interpretable steering directions than dense baselines while generalizing beyond cultural alignment.The broader evaluations include alignment tasks and benchmarks beyond culture.

2 Related Works

Prior activation-steering methods use dense or static directions that can be noisy, unstable, entangled, or insufficiently preference-optimized. YaPO combines SAE-based interpretability with learnable bi-directional preference optimization.

  • Activation engineering: Activation engineering freezes model weights and intervenes on hidden activations, but averaged activation methods can be noisy and unstable in alignment-critical generation.Earlier approaches include sentence-specific latent vectors, activation addition, and CAA.
  • Sparse activation steering: SAEs decompose activations into sparse, approximately monosemantic features, enabling SAS to provide interpretable and fine-grained control.SAS averages sparse activations from contrastive data.
  • Sparse activation steering: SAS does not optimize steering directions against preferences, limiting its effectiveness relative to preference-based learning.This distinguishes static sparse activation steering from YaPO’s trainable objective.
  • SAE-based steering and editing: ReFT-r1, RePS, and related methods explore dense, structured, or SAE-based intervention spaces with different objectives and constraints.The related-work landscape includes language-modeling objectives, reference-free bi-directional preference objectives, and SAE-space approaches.
  • Positioning of YaPO: YaPO unifies BiPO’s preference optimization with SAS’s interpretability by learning sparse steering vectors in SAE space.The paper positions this combination as producing disentangled, interpretable, stable steering with improved convergence and generalization.

3 Method

YaPO replaces dense activation-space steering with learnable preference-optimized vectors in a sparse SAE basis. The SAE transformation preserves the model representation while applying bidirectional behavioral perturbations through sparse codes.

  • Motivation: From Dense to Sparse Steering: Dense activation-space steering inherits neuron multi-semantic entanglement, producing noisy and unstable control signals in alignment-critical tasks.Contrastive prompt vectors can therefore misalign with actual generation behaviors.
  • Motivation: From Dense to Sparse Steering: SAEs map activations into sparse, interpretable features so steering vectors can be optimized along dimensions corresponding more cleanly to relevant semantic factors.The intended benefits are improved precision and interpretability.
  • Preference-Optimized Steering: BiPO injects a learned dense vector v into layer-L hidden activations, with d selecting positive or negative steering and λ controlling strength.The vector is learned from preferred and dispreferred responses using a bi-directional preference objective.
  • Preference-Optimized Steering in Sparse Space: YaPO encodes layer-L activations, adds a signed sparse steering vector, applies ReLU, decodes the result, and includes a residual correction for SAE reconstruction error.The learnable vector lies in sparse space, while encoder and decoder parameters remain those of a pretrained SAE.
  • Preference-Optimized Steering in Sparse Space: Bidirectional training increases the relative probability of preferred responses for d = 1 and enforces the reverse for d = −1.This symmetric objective sharpens alignment with the behavioral axis of interest.
  • Optimization: Only the steering vector v is updated during optimization; the LLM and SAE parameters remain frozen.The decoder projects the learned sparse direction back into the model’s hidden space.

4 Experiments

The experiments evaluate YaPO against dense and sparse steering baselines on cultural adaptation, using localized and non-localized prompts plus metrics designed to assess accuracy and robustness. YaPO converges substantially faster than BiPO and is reported as consistently strong across the experimental settings.

  • Experimental setup: YaPO is evaluated on Gemma-2-2B for cultural adaptation, with scalability to Gemma-2-9B deferred to Appendix D.The setup uses pretrained Gemma-Scope SAEs to enable sparse steering without training new SAEs.
  • Dataset and settings: Each cultural question has localized and non-localized forms, testing explicit country information against implicit inference from dialectal and situational cues.The paired design measures the performance drop when explicit country information is removed.
  • Evaluation metrics: PNLG normalizes the localized–non-localized performance gap by mean performance, while RCA uses their harmonic mean to reward accuracy and balance.Lower PNLG indicates better transfer across prompt settings; higher RCA indicates more robust cultural competence.
  • Baselines: The benchmark compares no steering, CAA, SAS, and BiPO against YaPO to separate the effects of sparse representations and preference optimization.CAA and SAS use averaging-based steering, while BiPO optimizes dense steering vectors.
  • Training dynamics: YaPO reaches loss below 0.1 in under 150 steps, whereas BiPO remains above 0.3 after 600 steps across the reported Egypt and Nepal scenarios.The paper attributes the faster convergence to disentangled SAE-latent features that provide cleaner gradients and more stable optimization.

5 Evaluation

YaPO is evaluated on multilingual cultural adaptation using MCQ and open-ended tasks, with analyses of robustness, convergence, steering sensitivity, general knowledge, and broader alignment-task generalization. Across these evaluations, YaPO is generally more consistent, stable, and balanced than dense or averaged-activation baselines, while preserving MMLU performance.

  • Evaluation setup: YaPO is evaluated against CAA, BiPO, SAS, and an unsteered baseline on multilingual cultural adaptation using MCQ and open-ended generation.The benchmark includes localized, non-localized, and mixed prompts, with results reported by language and country.
  • Multiple-Choice Questions: YaPO is the most consistent method across languages and prompt types, with especially pronounced gains on non-localized MCQs where cultural cues are implicit.BiPO is more variable and can underperform in low-resource or highly entangled settings.
  • Open-Ended Generation: YaPO yields the strongest Arabic open-ended gains, increasing average score from 2.97 to 3.37 in the non-localized setting, while BiPO leads Portuguese results.CAA substantially degrades Portuguese performance, whereas SAS remains close to baseline there.
  • Robustness metrics: YaPO achieves the highest RCA while maintaining among the lowest PNLG values across languages and tasks, improving cultural robustness without widening the explicit–implicit gap.RCA rewards balanced localized and non-localized performance, whereas lower PNLG indicates better transfer to implicit prompts.
  • Training dynamics: YaPO converges faster and more smoothly than BiPO across six cultural regions, reaching higher final accuracy while BiPO exhibits pronounced oscillations.The reported instability can overwrite previously correct behaviors, particularly in lower-resource settings.
  • Steering sensitivity: YaPO shows smooth, monotonic accuracy scaling over a wide range of λ values, unlike CAA and SAS, which can collapse beyond narrow operating ranges such as λ > 0.5.YaPO reaches highest accuracy at larger values such as λ = 1.5 or 2.0 without degradation.
  • General knowledge preservation: MMLU scores remain tightly clustered around the unsteered baseline across methods and prompt settings, indicating no significant degradation or inflation of general-purpose performance.The learned steering vectors primarily affect targeted alignment behaviors while broad knowledge capabilities remain intact.
  • Generalization: On hallucination, wealth-seeking, jailbreak, and power-seeking tasks, CAA has the highest average scalar score, while YaPO is typically second and less dependent on manual steering hyperparameters.The effective steering strength is absorbed into learned dimensions in YaPO and BiPO, giving YaPO more sparse degrees of freedom.

6 Conclusion

YaPO learns sparse, preference-optimized steering vectors in Sparse Autoencoder latent space and outperforms dense and baseline approaches across cultural and other alignment tasks.

  • YaPO learns sparse, preference-optimized steering vectors in the latent space of Sparse Autoencoders.
  • YaPO converges faster, is more stable, and is more interpretable than dense steering methods such as BiPO.
  • YaPO consistently outperforms BiPO and the baseline model on a multilingual cultural benchmark, especially for nonlocalized prompts.
  • YaPO generalizes beyond cultural alignment to hallucination mitigation, wealth-seeking, jailbreak, and power-seeking behaviors.

Limitations

The evaluation is broad but remains constrained by model coverage, SAE availability, and the cultural dataset’s treatment of within-country diversity.

  • Experiments cover only the Gemma-2 family, using 2B and 9B models, because of compute and time constraints.
  • The study does not evaluate additional architectures such as Llama-Scope 8B or Qwen models.
  • When no SAE is available, task-specific small SAEs or low-rank sparse projections remain future work.
  • The cultural dataset captures cross-country but not within-country diversity.

A Layer Discovery

Activation patching compares localized and nonlocalized prompts by transferring source representations into target passes, revealing layers that encode cultural localization signals.

  • Activation patching identifies which transformer layers contribute most strongly to cultural localization.
  • Representations at token i after block j depend on all preceding tokens under causal attention masking.
  • The analysis extracts localized-prompt representations and overwrites corresponding nonlocalized representations during the target forward pass.
  • Comparing perturbed and original predictions quantifies each layer’s contribution to culturally appropriate responses.
  • The study uses the last localized-prompt token and compares Egypt- and Morocco-specific answers with Western baselines.
  • Layer 15 is empirically identified as the most culturally relevant layer in Gemma-2-2B.

B Training Details

Training settings are reported in Table 6, with most configurations shared across model sizes and selected settings adjusted for memory and capacity constraints.

  • Most training settings are shared across model sizes.
  • Batch size, SAE configuration, and training time differ between the 2B and 9B models because of memory and capacity constraints.
  • Table 6 reports the training configuration and hyperparameters.

C Evaluation Results

The evaluation breakdowns show that YaPO maintains strong cultural-alignment performance across languages and settings while preserving general capabilities on MMLU. Results also expose trade-offs between cultural alignment and output naturalness.

  • MCQ and open-ended results: YaPO consistently delivers state-of-the-art performance, with the strongest accuracy across languages and cultural settings on MCQ.
  • MCQ and open-ended results: Steering generally improves performance over the unsteered baseline, with the strongest gains typically appearing in the Both setting.
  • MCQ and open-ended results: Method rankings remain broadly consistent with aggregated results, although country-level improvements are more heterogeneous in lower-resource settings.
  • RCA/PNLG analysis: Higher RCA can incur a PNLG cost, but several settings improve RCA while maintaining comparable or improved PNLG.
  • MMLU preservation: MMLU accuracy remains close to the unsteered baseline under culturally targeted steering.

D Scalability to other Models

Evaluation on Gemma-2-9B-it shows that steering remains effective at larger scale, though method rankings depend on task type and available headroom. The experiments use cultural, safety, and general-capability evaluations with consistent judgment protocols.

  • MCQ robustness at 9B: All steering methods improve Arabic MCQ performance over the unsteered baseline across most settings, but the stronger backbone reduces separation between methods.
  • MCQ robustness at 9B: BiPO most often attains the best average Arabic MCQ performance, while SAS, YaPO, and CAA provide comparable country- and setting-dependent gains.
  • Open-ended generation: SAS consistently yields the strongest Arabic open-ended scores, with BiPO and YaPO close behind and CAA less reliable for long-form generation.
  • General tasks and MMLU: Safety-suite performance improves modestly and consistently over baseline on average, with CAA slightly leading and SAS typically second.
  • General tasks and MMLU: MMLU remains essentially unchanged under steering at 9B, indicating targeted behavioral or cultural adjustments rather than broad capability shifts.
  • Evaluation protocol: The generalization tasks use identical scoring rubrics and LLM-judge prompts to compare BiPO and YaPO directly.

F.2 Dataset Statistics

The dataset provides broad, balanced coverage for evaluating cultural localization across languages, countries, and lived-experience topics. Its construction combines manually authored scenarios, model-assisted perspective generation, filtering, and contrastive packaging.

  • Dataset scale and coverage: 45,354 items cover five languages and fourteen countries, with approximately 1,372–1,607 questions per language–country variant.
  • Dataset scale and coverage: Localized and non-localized forms comprise 57.7% and 42.3% of the dataset, respectively, enabling estimation of the Localization Gap.
  • Topic coverage: The dataset spans 52 lived-experience topics, including meals, routines, family relations, etiquette, ceremonies, holidays, music, and idioms.
Loading 2601.08441v1…