Source-linked AI summary
Recurrence Is Not Enough: Causally Validating Multilingual SAE Translation Features in Gemma 2 and 3
Giang Son Nguyen, Nhi Ngoc-Yen Nguyen, Wray Buntine, Dung D. Le
TL;DR
The paper asks whether SAE translation features retain the same causal role across languages. It extends translation-feature discovery and intervention across multilingual settings in Gemma 2 and Gemma 3, finding that recurrence usually fails to predict causal transfer while one feature per model transfers robustly. The study therefore supports causal validation beyond recurrence, within the tested models, languages, and intervention design.
Problem
It is unclear whether SAE features found in one language context remain causally valid when prompt, source, or target languages change.
Method
The study reproduces translation-feature discovery and intervenes on recurring features across four discovery settings and 23 multilingual settings in Gemma 2 and Gemma 3.
Results
In both models, 22 features recurred across discovery settings, but only one per model consistently improved COMET when amplified and degraded it when ablated across 23 language configurations.
Takeaways & Limitations
Feature recurrence can overstate cross-lingual transfer, whereas the identified features provide evidence for a language-agnostic translation-initiation direction in these models.
Takeaways & Limitations
The identified features and intervention behavior are limited to Gemma 2 2B IT, Gemma 3 4B IT, their corresponding SAEs, and the tested discovery languages and settings.
Abstract
from arXiv · showhide
Sparse autoencoder (SAE) features are increasingly used to explain and steer language-model behavior, but it remains unclear whether a feature found in one language context plays the same causal role when processing prompts in another language. We study this question using translation-initiation features (Wu et al., 2026). We reproduce the SAE feature discovery method from Wu et al. in Gemma 2 and extend it to multilingual settings that vary prompt language, source language, and target language. We then test whether features that recur across settings affect translation behavior by amplifying or ablating their activations during inference. We also examine whether the method can be applied to Gemma 3. In both models, we observe an identical finding: although we can find more than 20 features that activate frequently across all discovery settings, causal validation shows that nearly all have small or inconsistent effects. In contrast, one feature -- Gemma 2's (L10, 5717) and Gemma 3's (L20, 2456) -- consistently improves COMET scores when amplified and degrades them when ablated across 23 language settings. These results show that feature recurrence can overstate cross-lingual transfer, while identifying a language-agnostic translation-initiation direction in Gemma 2 and Gemma 3.
1 Introduction
The study tests whether SAE translation features retain causal roles across multilingual settings and model families. Recurrence identifies many candidates, but causal validation finds robust transfer for only one feature per model.
- SAE explanations can be brittle, and multilingual settings may change whether a feature implements the same computation.
- The study varies prompt, source, and target languages while extending translation-feature discovery and intervention beyond English-centric evaluation.
- 22 recurring features were found across four discovery settings, but only (L10, 5717) showed robust causal transfer in Gemma 2.
- Gemma 3 reproduced the pattern: 22 recurring features appeared, yet only (L20, 2456) consistently improved or degraded under intervention across 23 language settings.
- The contributions frame multilingual feature recurrence as a weak proxy for causal transfer.
2 Related Work
Related work positions SAEs as tools for interpretable decomposition and intervention while emphasizing that mechanistic explanations can be brittle across languages and contexts.
- SAEs decompose model activations into sparse feature activations and decoder directions for interpretability and intervention.
- Prior work questions whether interpretability methods identify stable mechanisms rather than plausible but brittle descriptions.
- Multilingual models may share representations while also exhibiting language-specific computation or latent-language effects.
3 Method
The method reproduces a three-stage translation-feature pipeline, discovers candidates across multilingual settings, and evaluates interventions on held-out translation examples using COMET.
- 3.1 Notation: Feature notation uses (Layer, Feature Index), while settings use src2tgt_prompt to encode source, target, and instruction languages.
- The pipeline discovers candidates from boundary-token activations, filters them with PCA consistency, and intervenes by ablation or amplification.
- Four discovery settings progressively remove English-specific factors while varying prompt, source, and target languages.
- 23 language settings are constructed from six languages, using shared examples for discovery and held-out examples for intervention evaluation.
- 3.4 Dataset, model, and evaluation: Gemma 2 and Gemma 3 use corresponding 16k-width Gemma Scope SAEs, with translation quality measured by COMET.
- 3.4 Dataset, model, and evaluation: Four PCA-consistent features were obtained with the Gemma 3 small-L0 variant, an order of magnitude fewer than with big L0.
4 Results
Across both models, recurrence identified 22 shared features, but causal intervention isolated only one transferable translation-initiation feature per model. These features generalized across language settings, showed dose-dependent effects, and produced task-specific rather than generic multilingual improvements.
- Most overlapping features do not causally transfer: 21 of 22 shared features in each model formed a matched null with small, inconsistent, or oppositely signed intervention effects.They passed the same PCA Consistency filter but did not show the expected translation-switch signature.
- Most overlapping features do not causally transfer: Only one feature per model showed the expected causal signature: Gemma 2’s (L10, 5717) and Gemma 3’s (L20, 2456).Gemma 2 amplification gains ranged from +2.33 to +5.22, while Gemma 3 gains reached +2.23 and ablation ranged from −3.29 to −4.15.
- A single feature generalizes across multiple language settings: Gemma 3 amplification gains were consistently smaller than Gemma 2 gains, although significant in 18 of 23 settings, while ablation effects remained comparable.Gemma 3’s higher baseline was offered as one possible explanation for its smaller amplification headroom.
- Dose-response sanity check: The causal sign pattern persisted across intervention strengths: α = 0 hurt COMET, moderate amplification improved it, and effects typically peaked around α = 2–3 before saturation or reversal.The same pattern occurred in both models, indicating the effect was not tied to one coefficient.
- A single feature generalizes across multiple language settings: Across 23 settings, amplification improved COMET in every Gemma 2 setting and significantly improved it in 19 of 23 Gemma 3 settings.Ablation degraded translation in all Gemma 2 settings and 22 of 23 Gemma 3 settings; ar2en_ar was the sole Gemma 3 exception in both directions.
- Task specificity: On non-translation tasks, Gemma 2 intervention effects were small and inconsistent, supporting task-specific rather than generic multilingual gains.ROUGE-L changes ranged from −0.011 to +0.014, while entity-set micro-F1 changes ranged from −0.021 to +0.012.
- Translation initiation behavior: In Gemma 2, amplification reduced non-translation rates by up to 13.7 percentage points, while ablation increased them by up to 17.3 points.Gemma 3 showed the same pattern at smaller magnitudes.
5 Conclusion
The study concludes that feature recurrence is a weak proxy for causal transfer: 22 features recurred across discovery settings, but only one feature per model improved COMET across most of 23 language configurations. Dose-response results support a language-agnostic translation-initiation direction.
- Conclusion: 22 features recurred across four discovery settings in both models, yet only one per model improved COMET across the large majority of 23 language configurations.The transferable features were (L10, 5717) in Gemma 2 and (L20, 2456) in Gemma 3.
- Conclusion: Suppressing either transferable feature hurt COMET, while moderate amplification improved it across the four discovery settings.Figure 1 reports changes relative to the unmodified baseline at α = 1.0.
- Conclusion: The intervention effect persisted across changes in instruction, source, and target language, including fully non-English translation settings.This supports interpreting the transferable direction as language-agnostic within the evaluated models and settings.
Limitations
The study’s feature identities and intervention behavior are bounded by two models, selected discovery languages, individual-feature interventions, and COMET-centered evaluation.
- Scope and evaluation limits: The identified features and intervention behavior are limited to Gemma 2 2B IT and Gemma 3 4B IT with their corresponding Gemma Scope SAEs.Discovery used only English, Chinese, and Japanese, and did not cover all prompt–language combinations.
- Scope and evaluation limits: The study cannot fully disentangle prompt-language effects from translation-direction effects or assess whether the 21 non-transferring features contribute jointly.It evaluates features individually and does not test combined interventions.
- Scope and evaluation limits: Translation quality is measured primarily with COMET, while qualitative output interpretation relies on manual rather than systematic annotation.These choices may not fully capture adequacy, fluency, or output-type variation.
A.1 Reproducibility details
The experiments use deterministic decoding, fixed random seeds, and a single two-GPU machine to support reproducibility.
- Experimental configuration: All generations use greedy decoding with do_sample=False, max_new_tokens=128, and repetition_penalty=1.1.Temperature is therefore not used.
- Experimental configuration: The discovery/evaluation split and bootstrap resampling both use seed 42.This fixes the reported randomization choices.
- Experimental configuration: All experiments ran on one machine with two NVIDIA A100-SXM4-40GB GPUs.
B Interpretation and Qualitative examples
The evaluation classifies outputs as translations or non-translations using language identification, then compares baseline, amplification, and ablation. Across four discovery settings, amplification reduces non-translation rates while ablation increases them, with smaller effects in Gemma 3.
- Evaluation protocol: A 176-language fastText classifier labels outputs as non-translations when the detected language mismatches the target, the output is empty, or confidence is below 0.5.The evaluation reports the fraction of failed translations under baseline, amplification (α = 2), and ablation (α = 0).
- Results: In Gemma 2, baseline non-translation rates range from 15.8% to 36.0% across the four discovery settings.
- Results: Amplification reduces Gemma 2’s non-translation rate by up to 13.7 percentage points, whereas ablation increases it by up to 17.3 points in en2zh_en.
- Results: Gemma 3 shows the same directional pattern with smaller effects, consistent with its lower baseline non-translation rate of 5.3–10.6%.
C Non-core features in en2zh_en
The study tests whether features outside the four-way recurring intersection might also strongly initiate translation in en2zh_en. None matches the effect of (L10, 5717).
- Experiment: Features consistent in en2zh_en but absent from the four-way core intersection were amplified with α = 2.0.
- Results: +4.61 COMET points is the effect of (L10, 5717) in en2zh_en, while the largest positive effect among non-core features is only +0.65.
- Interpretation: Several non-core features substantially decrease COMET when amplified, so the four-way intersection did not omit another strong translation-initiation feature.
D Task-specificity evaluation details
The task-specificity evaluation tests whether amplifying (L10, 5717) improves multilingual generation beyond translation, using summarization and named entity recognition benchmarks. It measures task performance with ROUGE-L and entity-set micro-F1.
- Tasks: The evaluation tests amplification of (L10, 5717) on two non-translation tasks: summarization and named entity recognition.
- Summarization: Summarization uses 100 examples per language from XL-Sum in Chinese, Arabic, Russian, and Vietnamese, evaluated with ROUGE-L.
- Named entity recognition: NER uses 100 examples per language from WikiANN in Chinese, Japanese, Arabic, Russian, and Vietnamese, evaluated with entity-set micro-F1.
- Reporting: Tables 8 and 9 report baseline and amplified performance for the two tasks.
E chrF++ evaluation
The chrF++ evaluation provides a complementary test of intervention effects across recurrent features and 23 transfer settings. It broadly confirms that only the transferable feature in each model shows the expected translation-switch signature, while Gemma 3 has smaller amplification gains.
- Metric and scope: Corpus-level chrF++ is used as a complement to COMET for intervention outputs across the four discovery settings.
- Qualitative example: The qualitative example shows amplification producing a more direct translation and ablation producing English assistant-style responses instead of translation.
- Transferable features: Across 23 transfer settings, (L10, 5717) in Gemma 2 and (L20, 2456) in Gemma 3 are the only recurrent features showing consistent positive amplification and negative ablation.
- Intervention effects: Ablation consistently decreases chrF++ in both models, while amplification gains are clear in Gemma 2 but smaller and occasionally negative in Gemma 3.
- Model comparison: Gemma 3’s smaller amplification gains are attributed to its higher baseline, which leaves less headroom for surface-level improvement.