Source-linked AI summary
Translation Circuit Discovery and Targeted Optimization for Low-Resource Multilingual Translation
Atai Waris Khan, Abdullah Mazhar, Raghu Kumar, Md Shad Akhtar
TL;DR
Low-resource multilingual translation mechanisms remain insufficiently understood, particularly beyond English-centric and high-resource settings. TITAN identifies and optimizes translation-relevant circuits in Hindi–Urdu and Hindi–Sindhi; the paper reports BLEU gains and a shared semantic subspace rather than an English-centric pivot.
Problem
Mechanisms underlying low-resource multilingual translation remain insufficiently explored, while existing fine-tuning does not identify which components are causally responsible for translation behavior.
Source: §1 · p. 1Method
TITAN combines causal intervention, sparse feature decomposition, and targeted preference optimization to identify and optimize sparse translation-crucial circuits.
Source: §1 · pp. 1–2, §4 · p. 3Results
Takeaways & Limitations
For the studied low-resource Indic directions, translation circuits operate through a shared semantic subspace rather than an English-centric pivot representation.
Source: §1 · p. 2, §8 · p. 9Broader validation across additional languages, scripts, and domains remains necessary, and circuit discovery is computationally expensive.
Source: §9 · p. 9
Abstract
from arXiv · showhide
Recent multilingual large language models achieve strong translation performance, yet the internal mechanisms responsible for low-resource multilingual translation remain insufficiently understood. In this paper, we study translation circuits in multilingual decoder-only LLMs for Hindi-to-Urdu and Hindi-to-Sindhi (Devanagari). We propose TITAN, a mechanistic framework that combines causal intervention, sparse feature decomposition, and targeted preference optimization to identify and optimize crucial translation components in the transformer. Unlike conventional full-parameter fine-tuning, TITAN restricts optimization to sparse, causally important circuits, enabling scalable and parameter-efficient adaptation while updating fewer than 5% of model parameters. Experimental results show that TITAN achieves BLEU improvements of 33.75% on Hindi-to-Urdu and 45.86% on Hindi-to-Sindhi over the strongest multilingual baselines. Human expert evaluation further confirms substantial gains in adequacy and fluency across both language pairs. Additional analyses reveal that multilingual translation is implemented through sparse, structured circuits operating over a shared semantic subspace rather than through an English-centric pivot representation. Overall, our findings demonstrate that mechanistic circuit discovery can directly improve low-resource multilingual translation while providing interpretable insights into multilingual transformer behavior.
1 Introduction
Low-resource multilingual translation circuits remain underexplored, especially beyond English-centric and high-resource settings. TITAN identifies and optimizes causally important translation circuits, with reported quality gains and evidence of a shared semantic subspace in the studied Indic directions.
1 Introduction
- Mechanistic work on translation has remained focused largely on English-centric and high-resource settings, leaving low-resource multilingual translation mechanisms insufficiently explored.The introduction highlights South Asian languages’ lexical overlap, shared grammatical structure, script variation, and phonological divergence.Source: §1 · p. 1
- The study examines Hindi→Urdu and Hindi→Sindhi (Devanagari), which differ in script and, for Sindhi, also in phonology and lexical composition.The paper presents these complementary settings as enabling analysis of shared versus specialized translation mechanisms.Source: §1 · p. 1
- +33.75% BLEU on Hindi→Urdu and +45.86% BLEU on Hindi→Sindhi are reported alongside improvements in adequacy and fluency across both directions.The identified circuits are distributed across attention and MLP components, and optimization updates fewer than 5% of model parameters.Source: §1 · p. 2
- TITAN combines causal intervention, sparse feature decomposition, and targeted preference optimization to identify and optimize translation-crucial transformer components.It restricts optimization to sparse, causally important circuits instead of full-parameter fine-tuning.Source: §1 · pp. 1–2
- The studied Indic translation circuits operate through a shared semantic subspace rather than an English-centric pivot mechanism.The paper reports this finding for the studied low-resource Indic language pairs.Source: §1 · p. 2
2 Related Work
Prior work addresses low-resource translation, multilingual adaptation, and mechanistic analysis, but generally does not combine causal discovery of translation-specific components with their targeted optimization.
2 Related Work
- Low-resource NMT approaches include multilingual transfer, back-translation, transliteration, pivot modeling, and parameter-efficient language-specific adaptation, but focus on specialization rather than causal responsibility.The cited overview describes these approaches as efforts to improve translation quality or language-specific adaptation.Source: §2 · p. 2
- Prior multilingual interpretability work studies language-specific subspaces, language-aware parameters, and functionally relevant representations, whereas TITAN targets translation-specific circuits for optimization.The passage distinguishes TITAN’s focus from these prior approaches.Source: §2 · pp. 2–3
- Preference optimization has been used for multilingual generation and translation, but existing approaches generally optimize parameters without identifying causally responsible translation components.TITAN instead selectively optimizes translation-relevant components discovered through mechanistic circuit analysis.Source: §2 · pp. 2–3
3 Dataset
The dataset section describes two low-resource South Asian translation pairs and the corpora and contrastive data prepared for mechanistic analysis.
3 Dataset
- TITAN is evaluated on Hindi–Urdu and Hindi–Sindhi (Devanagari), complementary settings involving script divergence, lexical overlap, and phonological variation.The section summarizes corpus statistics for both directions.Source: §3 · p. 3, §3.1 · p. 3
3.1 Translation Corpora
- After preprocessing and deduplication, the Hindi–Urdu corpus contains 60K parallel sentence pairs and the Hindi–Sindhi corpus contains 35K.Both corpora are split into SFT, DPO, and contrastive analysis sets.Source: §3.1 · p. 3
3.2 Contrastive Translation Data
- For each language pair, 10K SFT sentence pairs are used to construct contrastive translations and counterfactual prompts that remove translation intent while preserving grammatical structure.Hindi–Sindhi sampling is stratified to cover Sindhi-specific implosive phoneme markers.Source: §3.2 · p. 3
3.3 Preprocessing
- Preprocessing includes Unicode normalization, duplicate removal, and sentence-length filtering, with script validation for Hindi, Sindhi, and Urdu.Sindhi-specific implosive markers are preserved for phonological circuit analysis.Source: §3.3 · p. 3
4 Methodology
TITAN identifies translation-relevant transformer components through causal analysis, then optimizes only those circuits. Its methodology combines contrastive prompts, sparse feature analysis, causal scoring, and targeted training.
4 Methodology
- TITAN builds on CIRCE to identify sparse components causally responsible for translation and restricts optimization to those circuits rather than updating all transformer parameters.Hindi–Urdu and Hindi–Sindhi provide complementary settings involving script divergence, lexical overlap, and phonological variation.Source: §4 · pp. 3–4
4.1 Contrastive Translation Construction
- Counterfactual prompts preserve grammatical structure while removing translation intent, helping separate translation-generation activations from general multilingual processing.Samples also include direction labels, vocabulary-overlap annotations, and phoneme markers, including Sindhi-specific implosives.Source: §4.1 · p. 4
4.2 CIRCE: Translation Circuit Identification
- CIRCE compares activations for valid and counterfactual prompts, intervenes along a translation-relevant direction, and retains components whose intervention substantially degrades translation.A P99 noise-floor criterion sets the operating threshold near 0.25%; discovered attention heads are categorized by routing behavior.Source: §4.2 · p. 4
4.3 Sparse Feature Decomposition
- SAEs trained on residual activations from CIRCE-identified layers expose disentangled translation features that are easier to interpret than raw activations.Restricting SAE training to causally relevant layers grounds the sparse features in translation circuitry rather than generic multilingual representations.Source: §4.3 · pp. 4–5
4.4 Feature-Level Causal Scoring
- Feature scoring combines language selectivity from semantically matched bilingual prompts with causal lift measured by intervention-induced translation-logit changes.Features with both high selectivity and positive causal lift are retained; low-scoring features are discarded.Source: §4.4 · p. 5
4.5 Targeted Translation Optimization
- CIRCE applies targeted SFT to identified components while freezing others; TITAN adds DPO using gold references as preferred outputs and SFT generations as rejected outputs.Training progresses from word- to phrase- to sentence-level translation, and KL regularization is used to preserve general language capabilities.Source: §4.5 · p. 5
5 Experiments and Results
TITAN outperforms multilingual baselines on Hindi–Urdu and Hindi–Sindhi, with especially large Hindi–Sindhi BLEU gains. Comparisons at matched parameter budgets and on additional language pairs also favor circuit-based optimization.
5 Experiments and Results
- Experiments compare multilingual MT systems and decoder-only LLMs under zero-shot, SFT, and DPO settings, alongside circuit-optimization and mechanistic analyses.The evaluated baselines include NLLB-200, IndicTrans2, Gemma-2, Qwen-3.5, LLaMA-3.1, and Mistral3.Source: §5 · p. 5
- +45.86% BLEU for H→S and +48.98% for S→H over the strongest baselines; TITAN consistently outperforms baselines across both language pairs.The comparison covers zero-shot, SFT, and DPO settings; TITAN results use Gemma-2 9B as the base model.Source: §5 · pp. 5–6, Table 2 · p. 6
- At the same 3.14% trainable-parameter budget, TITAN achieves 31.82 BLEU versus 26.37 BLEU for LoRA SFT+DPO, a +5.45 BLEU difference.The passage attributes the gain to circuit-based component selection rather than parameter count.Source: §5 · p. 6
- TITAN raises BLEU over DPO from 29.33 to 34.21 (Hindi→English), 28.71 to 31.94 (English→Hindi), 30.58 to 33.57 (English→French), and 33.09 to 36.40 (French→English).These are unseen high-resource tasks.Source: §5 · p. 6, Table 3 · p. 7
- On Hindi→Tamil and English→Irish, TITAN improves BLEU over DPO from 23.01 to 26.71 and from 29.52 to 33.19, respectively.The reported ordering across language pairs is SFT < DPO < CIRCE < TITAN.Source: §5 · pp. 6–7
6 Behavioral Pattern Analysis
The section tests the causal role and organization of TITAN-identified translation components, finding distinct routing, semantic-processing, and feature patterns across the studied Hindi–Urdu and Hindi–Sindhi directions.
6 Behavioral Pattern Analysis
- TITAN also outperformed SFT, DPO, and CIRCE on Hindi–Tamil and English–Irish, extending the reported gains beyond the primary Indic pairs.The figure caption describes performance on these additional typologically diverse low-resource tasks.Source: Table 4 · p. 7
6.1 Causal Validation via Progressive Knockout
- Ablating eight CIRCE-identified components reduced BLEU from 25.53 to 16.54 (35.2%), versus 25.53 to 25.41 (0.45%) for eight random components.For Hindi→Sindhi, removing a key MLP reduced translation accuracy from 100% to 54.4%; attention ablation degraded gradually, while MLP ablation caused sharper collapse.Source: §6.1 · p. 7
6.2 Attention Routing Behavior
- Attention routing progresses from structural anchoring and source-language identification to translation-direction routing and target-language commitment.Early heads attend to formatting boundaries and language identifiers; later heads shift toward direction routing and target-language commitment, with language-label anchoring prominent.Source: §6.2 · p. 8
6.3 MLP Representation Dynamics
- MLP representations show increasing source-language alignment through middle layers, but target-language alignment emerges mainly in final layers, with consistently negative English alignment.The observed dominant alignment is along shared Indic semantic directions rather than an explicit English pivot.Source: §6.3 · p. 8
6.4 Sparse Feature Behavior
- SAE features are layer-localized: target-language-specific features dominate earlier layers, while Hindi-specific features emerge more prominently later.Highly selective features are not necessarily strongly causal; translation behavior is associated with a sparse subset of high-lift features across source and target populations.Source: §6.4 · p. 8
7 Discussion
Expert judgments and metric analyses describe TITAN’s gains alongside residual error patterns. The reported improvements vary by language direction and include a trade-off between semantic fidelity and lexical reconstruction.
Expert Assessment
- For Hi–Ur, final TITAN adequacy rose from 36.7% to 76.7% and fluency from 53.3% to 86.7% relative to the Gemma-2 DPO baseline.The reported percentages are ratings of at least 4 on a 1–5 scale.Source: §7 · p. 8, Table 5 · p. 9
- For Hi–Sd, TITAN achieved 93.5% adequacy and 61.3% fluency, versus 51.3% and 25.8% for the Gemma-2 DPO baseline.Experts also observed better semantic preservation and more morphologically appropriate translations for technical terminology.Source: §7 · pp. 8–9, Table 5 · p. 9
Limitations
- Sindhi outputs sometimes retain untranslated Hindi loanwords or show minor Unicode inconsistencies, and long compound sentences can reduce fluency despite largely preserved adequacy.Source: §7 · p. 9
Error Analysis
- TITAN underperformed the strongest baseline only in TER for two Hindi–Urdu cases, while achieving stronger BLEU, ChrF2, and COMET on those directions.For U→H, BLEU rose from 25.78 to 27.01 and COMET from 77.37 to 81.28 despite higher TER; the largest gains were on shared-script Hindi–Sindhi directions.Source: §7 · p. 9
8 Conclusion
The conclusion presents TITAN as a parameter-efficient framework for multilingual translation and interprets the analyzed low-resource Indic circuits as sparse and structured.
8 Conclusion
- TITAN improves translation over multilingual baselines on Hindi–Urdu and Hindi–Sindhi while updating fewer than 5% of parameters.The paper concludes that the studied translation relies on sparse structured circuits within a shared semantic subspace rather than explicit English-centric pivot representations.Source: §8 · p. 9
9 Ethics and Limitations
The paper bounds its conclusions to the evaluated models and language settings, while outlining dataset construction, prompt and contrastive-pair design, validation, token roles, and experimental setup.
9 Ethics and Limitations
- Broader validation across additional languages, scripts, and domains remains necessary; causal findings are limited to evaluated models and language settings.The analysis focuses on identified circuits rather than the model’s complete multilingual representation space.Source: §9 · p. 9
A.1 Dataset Construction
- Hindi→Sindhi uses the same Devanagari script as Hindi but adds Sindhi-specific implosives and Persian-Arabic lexical influence; Hindi→Urdu tests a Devanagari-to-Nastaliq script boundary.The 10K sampled pairs per direction are used for contrastive prompts.Source: §A.1 · p. 11
A.2 Prompt Format
- The prompt’s trailing colon is the generation trigger and attention-analysis query position, while the arrow glyph is a single Gemma-2 SentencePiece token.The hyphen was found to fragment at the directive position and destabilize query-position alignment.Source: §A.2 · p. 11
A.3 Contrastive Pair Construction
- Each source sentence yields one positive prompt and four counterfactuals that differ only in the directive region, preserving token-position layout while substituting a targeted lexical component.The four perturbation types are illustrated with representative examples from both directions.Source: §A.3 · p. 11
A.4 Validation Protocol
- Validation preserves fluency for semantic obfuscation and target nullification, but assesses deeper structural perturbations by task disruption; all four types must yield zero held-out translation accuracy.Templates also require strong agreement between two annotators on whether the counterfactual removes the translation directive.Source: §A.4 · p. 11
A.5 Token Taxonomy
- Attention and SAE analyses assign IND tokens a structural scaffold role, SRC tokens as source-attention targets, and TGT as the readout site.Hindi→Sindhi samples with Sindhi-specific implosives are additionally tagged at SRC.Source: §A.5 · p. 11, Table 7 · p. 12
B Experimental Setup
- Baseline fine-tuned models use LoRA rank 16 and batch size 8 with Gemma-2 9B; TITAN additionally uses an SAE latent dimension of 14336 for feature discovery.SFT runs for 3 epochs and DPO for 2 epochs on NVIDIA A100 40GB GPUs.Source: B Experimental Setup · p. 12
C Causal Validation via Progressive Knockout
Progressive knockout shows that CIRCE-identified components are more causally important for translation than randomly selected components. Urdu MLP ablation produces a sharper early accuracy drop than random MLP ablation.
C Causal Validation via Progressive Knockout
- For Hindi→Urdu, removing the first key MLP reduces translation accuracy from 100% to approximately 87%, while random MLP ablation causes only marginal degradation.MLP ablation causes sharper early collapse, whereas attention-head ablation produces gradual decline.Source: C Causal Validation via Progressive… · p. 12
- Across both directions, knocking out CIRCE-identified components causes substantially greater degradation than random ablation, and the gap widens as more components are removed.Random components preserve most translation capability, while CIRCE-identified circuits approach near-chance performance.Source: C Causal Validation via Progressive… · pp. 12–13, Figure 2 · p. 13
D Attention Routing Behavior
Across Hindi→Sindhi and Hindi→Urdu, translation circuits combine language-aware attention routing with semantic processing in a shared Indic subspace. Sparse causally influential features, rather than an English pivot, support this account.
D Attention Routing Behavior
- Attention heads route from prompt boundaries and language identification toward translation-direction routing and target-language commitment, often attending to language indicators rather than source content.The strongest causal heads appear near the final generation layers.Source: D Attention Routing Behavior · p. 13
- Hindi→Urdu shows broader, more distributed mid-layer routing than Hindi→Sindhi, attributed to the added script-conversion complexity of Urdu Nastaliq generation.The findings suggest label-aware routing at prompt boundaries before downstream MLP semantic transformation.Source: D Attention Routing Behavior · p. 13
E MLP Representation Dynamics
- Source-language alignment rises through middle layers, while target-language alignment stays weak until sharply increasing near final generation layers.English alignment is consistently negative across nearly all layers; dominant alignment instead follows shared Indic, particularly Hindi-centered, semantic directions.Source: E MLP Representation Dynamics · p. 13
F Sparse Feature Behavior
- Across language pairs, target-language-specific SAE features dominate earlier layers, Hindi-specific features emerge later, and a middle transition zone shows neither language dominant.This pattern is consistent with the shared latent-space behavior in the MLP analysis.Source: F Sparse Feature Behavior · p. 14
- Strong selectivity alone does not imply causal influence: translation relies on a sparse subset of high-lift features across source- and target-language populations.The combined account assigns language-aware routing to attention and semantic transformation in a shared Hindustani latent subspace to MLP layers.Source: F Sparse Feature Behavior · p. 14