Bridge Routing Heads: Where Multilingual Multi-hop Reasoning Lives in LLMs

Motivation

• The study focuses on multi-step reasoning within each language, which two-hop questions probe by requiring an intermediate bridge entity before deriving the answer.The introduction distinguishes this capability from simple translation.Source: §1 · p. 1

The research question

• The identified BRH comprise a shared language-general pool alongside near-disjoint language-specific pools, rather than a single shared circuit or entirely separate circuits.The paper frames this pattern as its answer to whether successful multilingual chaining uses the same internal circuit.Source: §1 · p. 1

BRH Identification

• Language-specific BRH sets are nearly mutually exclusive across five languages, with mean Jaccard similarities of 0.017 for Llama and 0.057 for Qwen.The authors describe the resulting circuits as language-idiosyncratic.Source: §1 · p. 2

Causal Verification

• Ablating either BRH pool raises two-hop NLL, while amplification rescues failing passes through the language-specific pool intra-lingually and the general pool cross-lingually, without parameter updates.The authors interpret these findings as evidence that some failures reflect under-weighted routing rather than missing knowledge.Source: §1 · p. 2

Cross-model Analysis

• Llama concentrates chaining in a larger general pool, whereas Qwen relies on larger language-specific pools; the authors suggest this divergence may reflect differences in pretraining language mix and other factors.The paper names architecture, tokenization, and post-training among the possible factors.Source: §1 · p. 2

MI module roles

• Prior work describes MLPs as storing factual associations and attention heads as routing information, but whether this module-level division also governs reasoning circuitry remains open.The background also notes multilingual findings on language-specific output neurons and cross-language concept-space convergence.Source: §2 · p. 2

Multi-hop reasoning circuits

• Prior two-hop work identifies early bridge-entity resolution and a later second hop, while contemporaneous multilingual work studies failures and proposes fine-tuning; this paper instead analyzes successful chaining without parameter updates.The earlier account motivates separating heads by hop.Source: §2 · p. 2

Bridge Routing Heads

• Figure 1 depicts a two-hop query resolving a bridge entity, losing its signal when BRH routing is under-weighted, and recovering the answer after BRH amplification without training.It illustrates the proposed routing account of cross-lingual failures.Source: Figure 1 · p. 3

Multilingual multi-hop datasets

• Common multilingual extensions of multi-hop QA datasets translate English questions, potentially introducing translation artifacts and label noise, while paragraph-grounded benchmarks conflate parametric retrieval with in-context reading.The authors argue these features make such datasets unsuitable for studying reasoning circuits.Source: §2 · p. 3

3 Methodology

• A three-stage pipeline localizes the relevant module, identifies attention heads using a three-condition importance contrast, and verifies their role through ablation and amplification.The stages are applied to the multilingual two-hop dataset.Source: §3 · p. 3

3.1 Multilingual Two-hop Dataset

• The dataset composes Wikidata relations into two-hop queries across five languages without translation, retaining items with verified single-hop knowledge and failed shortcut controls.The source entity remains in English across languages, and the gold answer must identify the same Wikidata entity in all five.Source: §3.1 · pp. 3–4

3.2 Layer-wise Activation Patching

• Layer-wise activation patching substitutes source-language residual streams at the answer position into target-language prompts, separately for MLP and attention blocks, and scores output changes with KL divergence and logit difference.The setup uses direct QA prompts to emphasize parametric knowledge rather than in-context retrieval.Source: §3.2 · p. 4

3.3 Bridge Routing Head Detection

• The Bridge Routing Score contrasts first-hop and two-hop head importance against second-hop importance; mean-ablation filtering and Patchscopes verification then identify heads carrying the bridge entity.General candidates pass the scoring and ablation stages; language-specific candidates also undergo Patchscopes verification.Source: §3.3 · pp. 4–5

3.4 Verifying BRH

• The verification stage compares language-head overlap, ablates each BRH pool against layer-matched random heads, and amplifies BRH on incorrect target-language passes.In cross-lingual transfer, amplification targets general BRH on items answered correctly in English but incorrectly in the target language.Source: §3.4 · p. 5

4.1 Setup

• Experiments use two instruction-tuned 70B-class models with contrasting pretraining language mixes and compare ablations against layer-matched random head sets.The experiments draw from knowledge-verified items, with different subsets used for identification, ablation, and amplification.Source: §4.1 · pp. 5–6, Table 2 · p. 6

4.2 Module-level Localization

• Attention patching profiles diverge across language pairs, whereas MLP profiles remain tightly coupled, motivating head-level analysis of routing.The attention curves fan out across the mid-to-late layers; the MLP curves share a similar shape across pairs.Source: §4.2 · p. 6, Figure 3 · p. 6

4.3 BRH Identification

• Language-specific BRH pools are nearly disjoint: mean Jaccard similarity is 0.017 for Llama and 0.057 for Qwen.The general pool shares zero heads with each language-specific pool in both models.Source: §4.3 · p. 6
• Final BRH cluster around layers 14–38 in Llama and 47–57 in Qwen, aligning with the layer ranges where attention patching curves diverge.The Llama peak is dominated by general heads; Qwen’s later peak is dominated by Korean-specific heads.Source: §4.3 · p. 6, Figure 3 · p. 6, Figure 4 · p. 7

4.4 Causal Verification

• 39–89×: mean-ablating general BRH raises two-hop NLL above the unpatched baseline relative to the layer-matched random-head baseline.The NLL increase is 0.84–2.07 above the unpatched baseline, while random-head ∆NLL is at most 0.053.Source: §4.4 · p. 7, Table 4 · p. 7
• Language-specific BRH ablation raises NLL above random baselines in every model-language cell, and Patchscopes verifies bridge-entity encoding in surviving candidates.Qwen’s specific-pool effects are ∆NLL 0.50–0.93; Llama’s are 0.08–0.33, with each exceeding random by at least 8×.Source: §4.4 · p. 7, Table 4 · p. 7
• 41–51.7%: general-BRH amplification rescues cross-lingual failures in Llama, versus 12–34.8% in Qwen; specific-BRH rescue reaches 4.8–17.2%.Amplification improves accuracy monotonically with α for specific BRH, and the interventions use no parameter update.Source: §4.4 · pp. 7–8, Table 5 · p. 8, Figure 5 · p. 8

4.5 Model Comparison

• Llama has a larger general pool, while Qwen has larger language-specific pools; their amplification outcomes mirror this allocation difference.Mean general-BRH transfer is 45.3% for Llama versus 25.3% for Qwen; mean specific-BRH rescue is 11.9% for Qwen versus 6.1% for Llama.Source: §4.5 · p. 8

5 Discussion

• The authors frame amplification as surfacing already-known information rather than teaching the model, and relate BRH to a shared multilingual concept space.They propose general BRH route bridge entities through that workspace, while language-specific BRH handle recovery into the target output.Source: §5 · p. 8
• General-BRH output norms are nearly identical on success and failure passes, leaving routing-direction explanations compatible with the evidence.The norms differ by within ∼1%; the authors leave direct success–failure comparisons of output directions and in-context bridge readouts for future work.Source: §5 · p. 9
• The pool-allocation contrast cannot be attributed to pretraining language mix alone because architecture, tokenization, and post-training also differ.The authors call for controlled comparisons across additional model families to disentangle these factors.Source: §5 · p. 9

• Llama 3.1 70B and Qwen 2.5 72B both split BRH into a language-general pool and disjoint language-specific pools.The three-stage pipeline uses hop-differential scoring, mean ablation, and Patchscopes verification.Source: §6 · p. 9
• Ablating general BRH raises two-hop NLL by 39–89× the random-head baseline, while amplification rescues up to 51.7% of cross-lingual failures without training.Source: §6 · p. 9
• Some target-language failures reflect an under-weighted routing circuit rather than missing knowledge, motivating training-free multilingual model editing.Source: §6 · p. 9

• Set-level mean ablation averages over heads within a pool, so it can hide heterogeneous or opposing individual-head contributions.The authors identify per-head causal decomposition as a natural extension.Source: Limitations · p. 9
• Cross-model generality remains untested beyond two models that differ across several factors, including pretraining language mix, architecture, tokenization, and post-training.The authors call for additional model families.Source: Limitations · p. 9
• The five-language sample excludes low-resource languages with very different morphological systems and languages without sufficient Wikidata coverage.Source: Limitations · p. 9
• Patchscopes verification is permissive: a head qualifies after generating the gold bridge entity at least once per item across trials.It verifies heads that have already passed mean ablation rather than acting as a strict filter.Source: Limitations · p. 9
• The amplification sweep stops at α = 1.0, so reported rescue may be a lower bound, while higher scaling may also cost originally correct items.Source: Limitations · p. 10

A.1 Dataset Construction

• The dataset covers 17 Wikidata semantic entity categories and 29 relation types, with relation occurrence counts and entity distributions across chain positions tabulated.Source: §A.1 · p. 11, Table 6 · p. 11, Table 7 · p. 12
• Each two-hop triple yields five query variants, with multilingual examples illustrating the variants and a running chain from Stephen Harper through Canada to Ottawa.Source: §A.1 · p. 11, Table 8 · p. 12

A.2 Multilingual Extension

• Relation templates across five languages encode language-specific question forms, while the ASCII “Answer:” token keeps the readout position language-invariant.Instruction prefixes are localized per language.Source: §A.2 · p. 12

• MLP off-diagonal LD correlations are near chance: r = 0.02 (range −0.91–0.96) for Llama and 0.13 (−0.84–0.97) for Qwen.The reported MLP KL correlations are 0.989 for Llama and 0.936 for Qwen.Source: Figure 6 · p. 12, B Layer-wise Patching:… · p. 12
• MLP patches produce large distribution changes but highly variable, pair-dependent effects on the clean answer token, with late-layer LD dipping to approximately −3.5 for Qwen.Source: B Layer-wise Patching:… · p. 13
• Attention LD correlations are low—r = 0.18 for Llama and 0.34 for Qwen—while Qwen shows suppressive effects around layers 60–70 and Llama stays near zero.The reported correlation ranges are −0.75–0.93 for Llama and −0.57–0.92 for Qwen.Source: B Layer-wise Patching:… · p. 13

C BRS Contrast Sensitivity

• ρ = 0.962 for TH-only rankings, yet its candidate pools are 60% larger than the original: 305 vs. 191 Llama general heads and 291 vs. 175 Qwen heads.The TH-only pools still contain 84–89% of the original general pool as a subset.Source: C BRS Contrast Sensitivity · pp. 13–14
• The FH and SH terms narrow the candidate pool to heads important for first-hop inference but not for an externally supplied second hop.This separates importance ranking from chaining-specificity.Source: C BRS Contrast Sensitivity · pp. 13–14
• Alternative TH−SH and FH+TH−2SH contrasts yield Spearman ρ < 0.4 and retain only 26–35% of the original general pool.The table describes rank correlation, general pool size, and recall relative to the original contrast.Source: C BRS Contrast Sensitivity · p. 14, Table 10 · p. 14

D Source-Entity Localization Control

• ρ = 0.89–0.98 for Llama and 0.92–0.97 for Qwen across KO/ZH/JA/ES after source-entity localization.These are per-head ranking correlations against the original BRS over all 5,120 heads.Source: D Source-Entity Localization Control · p. 14
• Localization re-identifies 134 of 191 general-pool heads (70%) in Llama and 121 of 175 (69%) in Qwen.The passage attributes residual differences to heads near the top-N% intersection threshold moving in or out under perturbation.Source: D Source-Entity Localization Control · p. 14
• The language-general routing pattern is therefore not an artifact of keeping entity names in English.All other pipeline components are held fixed in this control.Source: D Source-Entity Localization Control · p. 14

E Amplification Cost

• At α = 1.0, the balanced set gains 90 items and loses 22 in Llama, and gains 57 and loses 24 in Qwen; correct-only accuracy falls to 89.5% and 87.3%, respectively.The balanced pool has a fixed composition, so these figures characterize the intervention trade-off rather than expected deployment gains when failing items are unknown.Source: E Amplification Cost · pp. 14–15
• English MMLU changes by at most 2.1 points across all ten model/head-set pairs; at α = 1.0, general-pool changes are −1.6 (Llama) and −0.5 (Qwen).The largest single drop, −2.1 points, occurs for Llama’s Japanese-specific set.Source: E Amplification Cost · p. 15

F Patchscopes Threshold Sensitivity

• Qwen retains more language-specific BRH than Llama at every Patchscopes threshold; the Korean ratio rises from 87/8 ≈11× at k = 0 to 49/1 ≈49× at k = 1.The dual-circuit pattern persists; Llama’s pools largely disappear at k = 1, while Qwen’s degrade more gradually, especially Korean.Source: F Patchscopes Threshold Sensitivity · p. 15

G Patchscopes Example Outputs

• Every representative continuation in the Patchscopes examples contains the gold bridge entity B; surface tokens from in-context examples are treated as benign leakage.The table rows pair target-language questions with English glosses.Source: G Patchscopes Example Outputs · p. 15, Table 14 · p. 17

H Head-Level BRS Landscape

• High-BRS heads form bands around layers 15–45 in Llama and 40–75 in Qwen, aligning with attention-divergence layers; final BRH fall within those bands.Figure 7 maps unthresholded per-head BRS across the 80-layer × 64-head grid, with layer 0 at the bottom.Source: H Head-Level BRS Landscape · pp. 15–16, Figure 7 · p. 17