Source-linked AI summary
Latent Mechanisms of Language Control in Multilingual Language Models
Ryo Mitsuhashi, Sabri Boughorbel, Majd Hawasly
TL;DR
Unintended code-switching complicates multilingual generation, motivating systematic study of mechanisms for language control. The paper compares three latent-selection methods in cross-layer transcoders and evaluates interventions across seven languages and two models. All methods effectively steer generation language, with FreqSel strongest overall and AnnSel providing interpretable selection.
Problem
Unintended code-switching makes multilingual models alternate inappropriately between languages, while systematic comparison of language-controlling mechanism discovery remains lacking.
Method
The paper compares activation-value, activation-frequency, and LLM-annotation methods for selecting language-specific latents in cross-layer transcoders, using two controlled benchmarks and targeted interventions.
Results
All three methods effectively steer generation language across seven languages and two models; FreqSel performs strongest overall, while AnnSel offers interpretable selection.
Takeaways & Limitations
The findings indicate that language control can rely on redundant, non-overlapping latent subsets rather than a single canonical language direction.
Takeaways & Limitations
The study is limited to CLTs from Gemma-2-2B and Qwen3-4B, seven predominantly high-resource languages, and two controlled code-switching tasks.
Abstract
from arXiv · showhide
Multilingual large language models can exhibit unintended code-switching -- unnecessarily alternating between languages during generation. We present a comparative study of three methods that identify language-controlling latents in cross-layer transcoders: activation value-based selection (ValSel), activation frequency-based selection (FreqSel), and LLM-generated latent annotation-based selection (AnnSel). To evaluate the efficacy of these methods in identifying language-controlling latents, we introduce two multilingual benchmarks that exhibit code-switching for fine-grained analysis of language steering across seven languages. Through targeted intervention experiments on Gemma-2-2B and Qwen3-4B, we find that all three methods effectively manipulate generation language, with FreqSel achieving the strongest overall performance, while AnnSel offering interpretable latent selection through explicit language annotations. A knock-out analysis suggests the methods select non-overlapping but each-functional latent subsets, indicating redundancy rather than a single canonical language direction. Code and data can be found at https://github.com/rm-3284/Latent-Mechanism-Multilingual.
1 Introduction
The paper addresses unintended multilingual code-switching by comparing three methods for discovering language-controlling latents and testing whether they identify shared or redundant representations.
- Unintended code-switching causes models to alternate inappropriately between languages during generation.
- Prior work identified neurons and latents that causally influence output language, motivating targeted interventions for fine-grained generation control.
- The study compares value-based ValSel, frequency-based FreqSel, and annotation-based AnnSel for identifying language-specific latents in cross-layer transcoders.
- The contributions include a controlled comparison, two benchmarks, seven-language intervention analysis, and an examination of redundancy across two model families.
2 Preliminaries
Prior mechanistic interpretability work studies language control through neurons, representation directions, and sparse features, while cross-layer transcoders provide a jointly trained dictionary spanning model depth.
- Language control has been studied through language neurons, PCA-derived language directions, and sparse autoencoder features.
- Cross-layer transcoders replace MLP blocks across all layers with a jointly trained sparse dictionary that captures features spanning model depth.
- The experiments use cross-layer transcoders trained for Gemma-2-2B and Qwen3-4B.
3 Manipulating Generation Language through Latent Interventions
The paper selects language-specific latents using activation magnitude, activation frequency, or annotations, then intervenes on them to steer generation toward a target language in controlled code-switching benchmarks.
- The study compares three latent-selection methods and four intervention strategies for steering generation toward a target language.
- Language-specific latent sets are defined within a shared CLT latent space using activation magnitude, activation frequency, or semantic annotation.
- Value-Based Selection (ValSel): ValSel ranks latents by language-specific activation magnitude and selects the top-K latents for each language.
- Frequency-Based Selection (FreqSel): FreqSel identifies latents that activate frequently for one language, requiring thresholded activation in examples and tokens while excluding other languages.
- Annotation-Based Selection (AnnSel): AnnSel traces attribution graphs, prunes high-importance paths, and filters contributing latents by annotations explicitly referencing the target language.
- Intervention Strategies: Interventions amplify target latents, ablate distractor latents, ablate distractor directions, or combine ablation with amplification.
- Evaluation Datasets: The Antonyms and Enumerations datasets place target-language outputs within context-language frames to create controlled code-switching evaluations.
4 Experiments
The experiments evaluate three latent-selection methods and intervention strategies on two controlled multilingual benchmarks across seven languages, using Gemma-2-2B and Qwen3-4B. All methods steer generation language, with FreqSel strongest overall and knock-out results indicating functional redundancy among latent sets.
- Datasets and evaluation: Antonyms creates controlled code-switching by requesting a target-language antonym from a different-language context, while Enumerations evaluates open-ended target-language continuation.The evaluation records target-token logit changes for Antonyms and length-normalized target-sequence logprob changes for Enumerations.
- Intervention strategy effectiveness: Zero+Amp is the most effective intervention, combining distractor-language zero ablation with target-language amplification.It outperforms 1L+Amp, which uses single-layer directional ablation of the distractor language; the authors suggest related-language latent overlap may weaken direction ablation.
- Selection method efficacy: All three selection methods are effective across models and settings, with FreqSel producing the greatest total absolute logit and logprob change and especially strong gains in Chinese and Korean.AnnSel performs decently across languages but is weaker in Korean, potentially because fewer Korean latents were extracted.
- Latent redundancy: The knock-out experiment amplifies one method’s latents while ablating the residual-stream projection associated with another method’s direction to test independent control.Generally positive aggregate changes show robustness, while asymmetry indicates that the latent sets are not completely independent.
- Latent redundancy: Different methods identify overlapping yet non-identical language representations: same-language latent-set intersections are non-empty, while language-direction similarities vary.These observations support functional redundancy rather than a single canonical language direction.
5 Conclusion
The study compares three methods for identifying language-specific latents in cross-layer transcoders and finds that all can steer generation language across seven languages. FreqSel performs best overall, while AnnSel adds interpretability through explicit language annotations.
- The study compares value-based ValSel, frequency-based FreqSel, and annotation-based AnnSel for identifying language-specific latents in cross-layer transcoders.
- All three methods effectively steer generation language on the controlled Antonyms and Enumerations tasks across seven languages.
- FreqSel achieves the strongest overall performance, particularly for Asian languages.
- AnnSel provides interpretable latent selection through explicit language annotations.
- Future work should test whether these findings generalize to larger models, more languages, and real-world code-switching scenarios.
6 Limitations
The study’s evidence is constrained by its models, languages, benchmarks, annotation quality, and incomplete computational-cost evaluation.
- Experiments used CLTs from only two small models, Gemma-2-2B and Qwen3-4B, so findings may not generalize to other architectures or standard single-layer SAEs.
- Evaluation covered seven predominantly high-resource languages and two controlled code-switching tasks, while real-world scenarios may present different challenges.
- AnnSel depends on Neuronpedia annotation quality, with inconsistent annotations evidenced by only 7 Korean latents versus 45+ for other languages.
- Computational costs were not systematically evaluated, although AnnSel requires additional circuit tracing compared with statistics-based methods.
Potential Risks
The paper discusses potential misuse of language-steering interventions and describes the controlled pipeline used to identify and test language-specific latents.
- Language-steering interventions could be repurposed to force a specific language or suppress a language for deceptive purposes.
- The authors assess practical risk as low because the interventions require direct access to CLT weights and latent activations and were demonstrated only on small models in controlled settings.
- AnnSel constructs attribution graphs, prunes important paths, and selects latents whose annotations reference the target language.
- Attribution graphs explain top predicted logits using n = min(5, m), where m is the smallest integer whose cumulative probability reaches 0.95.
- Graph pruning retains nodes explaining 80% of output-logit influence and edges explaining 98%.
- AnnSel filters feature descriptions for language or relevant country names, then keeps features with frequency at least 0.1 of the maximum listed frequency.
- Keyword matching can select semantically incorrect latents, such as labeling a “Non-English characters” feature as English.
- ValSel’s activation pattern is highest in the original context and close to zero for modified-noun contexts.
C.2 FreqSel and text-generation experiment
FreqSel identifies language-specific latents using activation patterns and steers unconditional generation toward target languages, though outputs may be unnatural.
- FreqSel method: FreqSel selects language-specific latents using activation maxima determined from multilingual corpora.Its steering rule adds a scaled maximum-activation latent to the representation.
- Text generation: Unconditional Gemma-2-2B generation is steered toward target languages using the selected features.Examples use a [BOS] prompt and top-p sampling.
- Text generation: The generated sentences are not always perfectly natural despite being directed toward the target languages.
- Latent counts: ValSel extracts 50 latents by design, whereas FreqSel and AnnSel return variable numbers related to CLT training data.
E Computational Cost of Each Method
The study compares method-specific computational costs and notes that AnnSel is more expensive because it adds circuit tracing and graph pruning, while Qwen’s larger CLT increases costs.
- Cost measurement: Energy costs are reported cumulatively in GPU-hours and kWh for processing 100 sentences across seven languages.The estimate uses an L40 GPU and treats the resulting energy consumption as a likely lower bound.
- Method comparison: AnnSel requires more computation than ValSel and FreqSel because it adds circuit tracing and graph pruning.
- Model comparison: Qwen’s additional cost reflects a larger CLT, with 163,840 features per layer across 36 layers versus Gemma’s 16,384 features across 26 layers.
- Sensitivity settings: The latent budget is fixed to the smallest count identified across methods, with German AnnSel latents expanded to 50 by frequency ranking.
- Sensitivity settings: ValSel uses k as its only hyperparameter, and the study sets k = 50 after observing larger average total changes with more selected latents.
F.3 FreqSel sensivitity analysis
FreqSel sensitivity analysis uses three thresholds, while detailed intervention results compare method behavior across language pairs, models, and benchmark settings.
- FreqSel sensitivity analysis: FreqSel varies cross-lingual threshold T, active-example percentage N, and active-token percentage M, using defaults T = 0.8, N = 98, and M = 10.
- Zero+Amp results: The Antonyms logit-margin tables compare before-versus-after intervention values by context language rows and adjective-language columns.Smaller post-intervention margins are desired; red cells indicate failures.
- Zero+Amp results: FreqSel is strongest overall, while ValSel is fragile with Chinese contexts and AnnSel is fragile in Korean.AnnSel identifies only seven Korean latents, whereas FreqSel is described as most robust there.
- Zero+Amp results: French is distinctive: ValSel and FreqSel fail for some contexts, while AnnSel shows better robustness.
- Continued generation: Held-out FLORES+ sentences are used with top-p = 0.5 and Zero+Amp to test whether discovered latents work beyond the evaluation datasets.
M Additional Redundancy Analysis
The redundancy analysis tests whether methods identify equivalent latent directions and examines language-referencing annotations as an overlap proxy.
- Direction comparison: Per-method representative layer-wise activation directions are compared with pairwise cosine similarity to assess latent-set equivalence.
- Annotation analysis: Neuronpedia annotation overlap is estimated as the percentage of language latents whose annotations explicitly refer to that language.
- Annotation analysis: AnnSel reaches 100% explicit language-reference annotations by design, while ValSel and FreqSel vary substantially.
- Analysis scope: The analysis includes the Enumerations dataset alongside layer-wise direction and annotation comparisons for Gemma-2-2B and Qwen3-4B.