PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Low-Resource Language Reasoning via High-Resource Language Feature Transfer

Minju Song, Hyeon Hwang, Junhyun Lee, Jaewoo Kang

arXiv:2608.30462v1cs.CLcs.AI

TL;DR

The paper investigates why semantically equivalent reasoning tasks perform differently across languages and whether HRLs more reliably elicit useful latent computations. It identifies contrastively selected sparse features from successful HRL traces, steers LRL inference with their residual-stream directions, and reports partial gap closure without parameter updates or translation. The results support feature-mediated transfer as an interventional route within the studied scope.

  • Problem

    The paper asks whether cross-lingual reasoning gaps reflect different elicitation of internal task computations rather than only differences in language surface form or data exposure.

  • Method

    The framework uses sparse autoencoders and contrastive HRL–LRL trace comparisons to select task-relevant features, then injects their residual-stream steering direction during LRL inference.

  • Results

    25–33% of the gap to the reference language is closed across two models, three benchmarks, and four low-resource target languages, with no parameter updates or translation.

  • Takeaways & Limitations

    Intervention tests support interpreting the selected sparse features as a mechanistic handle for accessing computations more reliably elicited in the reference language.

  • Takeaways & Limitations

    Whether the approach extends to other model architectures, reasoning domains, and truly extremely low-resource languages remains to be verified.

Abstract

from arXiv · show

Large language models exhibit substantial performance variation across languages, even when solving semantically equivalent tasks. Existing analyses often treat this phenomenon as an observational disparity caused by differences in pretraining data, tokenization, or benchmark coverage. We study a complementary hypothesis: high-resource languages (HRLs) may more reliably elicit latent computations useful for task-specific (i.e. mathematical) reasoning, while lower-resource languages (LRLs) may under-activate those computations despite expressing the same task. To test this hypothesis, we introduce a mechanistic intervention framework for identifying and transferring task-relevant sparse latent features across languages. Using sparse autoencoders over residual-stream activations, we isolate features enriched in successful HRL task-specific reasoning while filtering out source-language and generic-generation features. We then construct steering directions from these features and inject them during LRL inference. The resulting interventions test whether the selected features are functionally involved in the observed reasoning gap: suppressing them should impair source-language reasoning, while activating them should partially recover target-language reasoning beyond random and non-task controls. Our framework reframes some cross-lingual reasoning gaps as failures of mechanism elicitation rather than capability absence, and offers a causally testable route to feature-mediated transfer without translation, fine-tuning, or changing the user-facing language.

1 Introduction

The paper asks whether language changes the internal reasoning mechanisms a model elicits, not only the surface form of equivalent inputs. It proposes identifying task-relevant features from successful HRL reasoning and transferring them to LRL inference through causal interventions.

  • Motivation: LLM mathematical reasoning is substantially stronger in HRLs than in equivalent LRL prompts, a disparity commonly linked to uneven training and evaluation data.English dominates pretraining data, while low-resource languages receive a tiny share, and this imbalance accompanies reasoning-accuracy differences.
  • Problem: The central question is whether language changes which internal reasoning mechanisms a model uses rather than only changing input surface form.The hypothesis is that HRLs more reliably elicit computations such as symbolic decomposition or stepwise arithmetic reasoning.
  • Approach: The framework identifies task-relevant sparse features from successful HRL reasoning using sparse autoencoders and contrastive filtering against generic HRL behavior.Contrastive selection is intended to exclude features encoding language identity or broad generation patterns.
  • Approach: The selected features form a residual-stream steering direction injected during LRL inference while leaving the prompt unchanged.The intervention amplifies computations more strongly expressed in successful HRL reasoning and tests whether they improve LRL inference.
  • Validation: Causal validation tests activation, suppression, and matched random or excluded-feature controls in a fixed trained model.The design distinguishes observational feature association from interventional evidence of functional involvement.

2 Preliminaries

The preliminaries define sparse autoencoder representations of residual-stream activations and the notation for comparing HRL and LRL reasoning traces. They also specify interventions that test partial sufficiency, functional necessity, and specificity within a fixed model.

  • 2.1 Sparse Autoencoders (SAEs): Sparse autoencoders encode decoder-only model activations into sparse latent representations and reconstruct them through linear decoder directions.The encoder applies a sparsity-inducing nonlinearity, while each decoder column represents a latent feature’s residual-stream direction.
  • 2.2 Problem Setup and Notation: The setup analyzes chain-of-thought activations at a chosen residual-stream layer for a reference HRL b and target LRL a.A benchmark produces language-specific reasoning traces across token positions, whose latent feature activations are analyzed.
  • 2.2 Problem Setup and Notation: The goal is to identify features associated with successful HRL reasoning and use them to steer behavior during LRL problem solving.The intended transfer is from reference-language reasoning traces to target-language inference.
  • 2.2 Problem Setup and Notation: Feature selection from naturally generated traces is observational and does not by itself establish causality.Causal claims are restricted to interventions within the fixed trained model.
  • 2.3 Interventional Causal Effects: The intervention directly modifies residual-stream activations during the forward pass and evaluates final-answer correctness or a logit-based answer score.The notation represents the model outcome after adding an intervention vector to an internal activation.
  • 2.3 Interventional Causal Effects: A feature set is partially sufficient when activation improves LRL reasoning, functionally necessary when suppression degrades HRL reasoning, and specific when controls lack the same effect.These criteria operationalize the paper’s intervention tests.

3 Method

The method identifies sparse features associated with successful HRL reasoning by contrasting matched HRL successes with LRL failures, then transfers those features through residual-stream steering during LRL generation.

  • Task-Relevant Sparse Latent Feature Identification: The framework contrasts reasoning traces on identical problems where the HRL succeeds and the LRL fails.
  • Task-Relevant Sparse Latent Feature Identification: For each contrast pair, it records the strongest-activating SAE feature at every generation token and forms per-response feature sets.
  • Task-Relevant Sparse Latent Feature Identification: The candidate pool retains features appearing in HRL-correct traces but absent from matched LRL-incorrect traces.
  • Task-Relevant Sparse Latent Feature Identification: Rank-window filtering removes highly frequent generation-pattern features and retains candidates within a predefined rank window.
  • Transfer via Steering at the Residual Stream: For selected features, steering coefficients equal the difference between mean SAE activations on correctly solved HRL and LRL traces.
  • Transfer via Steering at the Residual Stream: During target-language inference, the method shifts selected latent features, decodes the difference into residual space, and adds the same vector at every generation step while leaving prompt prefill unchanged.

4 Experiments

The experiments evaluate the intervention with two instruction-tuned models, multilingual reasoning benchmarks, several HRL–LRL pairings, and native-language chain-of-thought prompts.

  • Models and Representations: The study uses two instruction-tuned models with pretrained layer-20 SAEs and applies all steering interventions at the layer-20 residual stream.
  • Benchmarks and Languages: Evaluation covers MATH500, multilingual MMATH, MGSM, and MMLU-ProX Psychology, spanning symbolic reasoning, grade-school mathematics, and undergraduate psychology.
  • Benchmarks and Languages: English and Spanish serve as HRL reference languages, while Korean, Thai, Swahili, and Vietnamese serve as LRL target languages.
  • Evaluation Setup: Each problem is presented in one language with a native-language chain-of-thought prompt, using translations from publicly available multilingual benchmark releases where applicable.
  • Feature Selection: Feature pools are built from reference-correct and target-incorrect problems, then filtered by discarding the top 50% and retaining features between the 50th and 90th occurrence percentiles.
  • Evaluation Metrics: Accuracy follows each benchmark’s native evaluation protocol, while recovery rate measures how much steering closes the target–reference accuracy gap.

5 Discussion

The discussion presents contrastive sparse-feature interventions as causal tests of cross-lingual reasoning transfer, showing target-side gains, source-side degradation under suppression, and largely language-agnostic feature content.

  • Intervention tests: Contrastively selected features are evaluated as candidate intervention features for transferring task-relevant computations from reference-language reasoning to target-language inference.The framework distinguishes feature discovery from causal validation through three intervention tests.
  • Partial sufficiency: Activating pair-specific feature sets improves target-language accuracy over the corresponding low-resource baseline for Thai, Korean, and Vietnamese.The gains are largest for Korean and Vietnamese.
  • Partial sufficiency: Steering largely preserves target-language generation rather than generally switching outputs into the reference language.Language-identification results are reported across models, datasets, and reference–target pairs in Appendix B.
  • Functional necessity: Suppressing the same pair-specific features during English inference produces drops from the English baseline, supporting their functional involvement in reference-language reasoning.The ablation scores remain on English, with columns identifying the target language used to construct each feature set.
  • Specificity: Random, top-k frequent, and excluded-feature controls do not reproduce the consistent improvement pattern of the contrastively selected set.Negative steering also reduces target-language accuracy, distinguishing the signed activation difference from a generic magnitude increase.
  • Language-agnostic features: 25 of 28 selected features activate in both English and Korean, while three are English-token specific; shared examples encode quantity, requirement, and equality.Feature 13089 peaks at 28.0 in Korean versus 63.6 in English, consistent with weaker low-resource activation.

6 Related Work

Related work spans multilingual reasoning disparities, mechanistic interpretability with sparse autoencoders, and residual-stream steering, while this method selects SAE features contrastively for cross-lingual transfer.

  • Multilingual reasoning and cross-lingual transfer: Prior multilingual reasoning work documents stronger reasoning in high-resource languages and commonly attributes the gap to imbalanced pretraining and instruction-tuning data.Earlier remedies often rely on additional multilingual training.
  • Mechanistic interpretability and SAEs: Mechanistic interpretability decomposes language models into interpretable units and validates behavioral hypotheses with interventions such as activation patching and causal mediation.Sparse autoencoders are part of this broader effort to decompose model internals.
  • Activation and feature steering: Activation steering typically adds residual-stream vectors derived from contrasting prompts, with later work extending scale, target behaviors, learned vectors, and input conditioning.SAE-level steering instead clamps or edits individual interpretable features.
  • Activation and feature steering: This method differs by identifying reasoning-relevant features contrastively from paired reference-correct and target-incorrect traces to bridge a cross-lingual performance gap.It uses a selected feature set rather than a single labeled concept or hand-chosen latent.

7 Conclusion

The conclusion frames cross-lingual reasoning gaps as partly related to latent-computation elicitation and reports feature steering as a no-update, no-translation intervention with partial gap closure.

  • Conclusion: Across two models, three benchmarks, and four low-resource target languages, the intervention closes 25–33% of the gap to the reference language.The result requires no parameter updates, translation, or change to the user-facing language.
  • Conclusion: Three intervention tests provide evidence that selected features are functionally implicated in the observed cross-lingual reasoning gap.The tests assess partial sufficiency, functional necessity, and specificity.
  • Conclusion: The findings suggest that part of the gap reflects under-elicitation of computations the model already possesses rather than their absence.Feature-level intervention is presented as complementary to translation- and training-based remedies.

Limitations

The evaluation covers two instruction-tuned models, three benchmarks, and four low-resource target languages, but broader model, domain, and language generalization remains unverified. The method also depends on pretrained single-layer SAEs, and cross-task transfer is not explicitly modeled.

  • The study evaluates two instruction-tuned models across three benchmarks and four low-resource target languages.
  • Generalization to larger or differently architected models, other reasoning domains, and extremely low-resource languages remains to be verified.
  • Feature resolution and steering effectiveness are bounded by the quality of the externally trained SAE at layer 20.Polysemantic or under-trained latents may dilute the signal isolated by contrastive selection.
  • The experiments primarily transfer features across languages within the same task, leaving cross-task transfer unmodeled.When task-specific features transfer across benchmarks remains an open question.

A Implementation Details

The implementation applies sparse-feature steering at residual-stream layer 20 in two open instruction-tuned models, using greedy native-language decoding and language-identification checks. Layer comparisons test whether the intervention depends on this primary layer.

  • A Implementation Details: The experiments use Gemma-2-9B-it and Qwen2.5-7B-Instruct, with all interventions applied to the layer-20 residual stream.
  • A Implementation Details: The models use publicly released or specified layer-20 JumpReLU sparse autoencoders with widths 16,384 and 65,536.
  • A Implementation Details: Experiments run with Python 3.11, PyTorch 2.4.1, CUDA 12.1, transformers 4.51.0, and sae_lens 5.5.2.
  • A Implementation Details: Jobs run on four NVIDIA H100 80 GB GPUs, using one GPU per job.
  • A Implementation Details: Greedy decoding allows up to 1,024 new tokens, with native-language prompts requesting step-by-step reasoning and boxed final answers.
  • A Implementation Details: GlotLID measures intended target-language output both per sentence segment and per complete response.
  • A Implementation Details: Steering changes the target-language proportion by less than 1.5 percentage points in 37 of 40 settings, averaging 0.8 points.
  • A Implementation Details: Layer-20 steering is compared with layers 4, 12, 34, and 38 on Gemma-2-9B MGSM experiments using English as reference.

D Comparison with Related Steering Method

The paper compares its sparse-feature residual-stream intervention with dense hidden-state steering through CAA. Its SAE-based method combines stronger accuracy with feature-level interpretability in the reported settings.

  • D Comparison with Related Steering Method: CAA steers dense hidden-state differences, whereas the proposed method steers sparse features associated with reasoning-relevant concepts shared across languages.
  • D Comparison with Related Steering Method: SAE-based steering achieves higher accuracy than CAA in 11 of 12 evaluation settings while retaining individually identifiable features.

E Generalization to Unseen Target Languages

The intervention generalizes to Bengali and Telugu and can partially transfer feature sets from other target-language pairs. Pair-specific extraction remains strongest, consistent with target languages under-eliciting partly different reasoning features.

  • E Generalization to Unseen Target Languages: Pair-specific steering raises Bengali accuracy from 75.6% to 81.2% and Telugu accuracy from 74.4% to 75.6%.Target-language generation is preserved in these additional-language experiments.
  • E Generalization to Unseen Target Languages: Feature sets extracted from EN→KO and EN→TH both improve Bengali and Telugu over baseline, although pair-specific extraction remains strongest.
  • E Generalization to Unseen Target Languages: The results suggest that target languages under-activate partly different subsets of reasoning features, making pair-specific extraction more effective.
Loading 2608.30462v1…