LangBridge: Multilingual Reasoning Without Multilingual Supervision

1 Introduction

• Prior adaptation by continual training requires target-language corpora, making it challenging to scale across many languages; specialized models such as MetaMath often use English datasets.The paper also notes that reasoning LMs tend to underperform on low-resource languages, which are represented as long-tail knowledge.Source: §1 · p. 1
• LANGBRIDGE adapts reasoning-specialized language models without multilingual training by adding trainable parameters between a multilingual encoder and the target model, using English data.The approach is presented as zero-shot cross-lingual transfer at test time.Source: §1 · pp. 1–2
• 53.5% average accuracy on MGSM for MetaMath-13B with LANGBRIDGE, versus 40.5% without it, matching PaLM-540B’s 51.3%.The comparison is reported for MetaMath-13B leveraging an mT5-XL encoder.Source: §1 · p. 2
• The authors report gains across mathematical reasoning, code completion, and logical reasoning, plus improvements on linguistically demanding Big-Bench Hard subtasks and XCOPA.They hypothesize that language-agnostic multilingual representations help the target model grasp input semantics; PCA and qualitative analyses support this hypothesis.Source: §1 · p. 2

2.1 English-centric Language Models

• English-centric reasoning models adapted on English domain-specific data inherit limited low-resource-language proficiency, creating a performance discrepancy that LANGBRIDGE aims to narrow.Source: §2.1 · p. 2

2.2 Zero-shot Cross-lingual Transfer

• Multilingual models transfer across languages after high-resource-language finetuning; LANGBRIDGE similarly enhances reasoning across languages while adapting with English data alone.Source: §2.2 · p. 2

2.3 Aligning Pretrained Representations

• Prior alignment work connects visual encoders to language models, while a concurrent two-LM approach uses multilingual supervision for alignment, unlike LANGBRIDGE.Source: §2.3 · p. 2

3.1 Hypothesis

• The hypothesis is that aligning a multilingual encoder to a language model’s space lets the model understand supported languages without training on many languages.The method tests this by aligning with English corpora and evaluating on non-English inputs.Source: §3.1 · p. 2
• A multilingual encoder and added linear layer are aligned to the target model using English data; the target model stays frozen, while encoder training depends on whether the target LM is pretrained or finetuned.The encoder is trainable for pretrained LMs and frozen for finetuned LMs.Source: Figure 2 · p. 3

3.2 Model Architecture

• LANGBRIDGE maps multilingual encoder final hidden states to the target LM’s soft prompts with a single linear layer and appends a trainable end-of-sequence token.Padding-derived prompt tokens are masked for the target model.Source: §3.2 · p. 3
• The target-token likelihood is conditioned on encoder input tokens interpreted as prefix tokens; L denotes target sequence length and x_i its i-th token.Source: §3.2 · p. 3

4.1 Overview

• LANGBRIDGE is evaluated on mathematical, code-completion, logical, and commonsense reasoning using multilingual or translated benchmarks.The main experiments include MGSM, translated HumanEval, and BBH-BN; commonsense results are in Appendix A.Source: §4.1 · p. 3

4.2 Mathematical Reasoning

• For mathematical reasoning, MGSM covers 11 languages; pretrained models use cross-lingual CoT exemplars, while finetuned models are evaluated zero-shot.Llemma and MetaMath are adapted using samples from their respective training datasets; Table 1 separates pretrained and finetuned results.Source: §4.2 · p. 4, Table 1 · p. 5

4.2.2 Results

• The MGSM results show LANGBRIDGE improves Llemma and MetaMath especially in underrepresented languages, but may reduce high-resource performance.MetaMath-LB is competitive with MathOctopus despite English-only math training, and LANGBRIDGE remains robust across MGSM's 11 languages, including Telugu.Source: §4.2.2 · pp. 5–6
• LANGBRIDGE models vastly outperform similar-sized multilingual baselines on mathematical reasoning, supporting their use for low-resource-language math.The MGSM table reports accuracy by overall, high-resource, and underrepresented-language groups and compares LANGBRIDGE with original models.Source: Table 1 · p. 5, §4.2.2 · p. 6

4.3.1 Experimental Setup

• The code evaluation translates HumanEval instructions into five underrepresented languages, anonymizes function names, and compares LANGBRIDGE with original checkpoints and baselines.Table 2 reports zero-shot Pass@1 using greedy decoding on anonymized HumanEval and HumanEval-MT.Source: §4.3.1 · p. 6, Table 2 · p. 7

4.3.2 Results

• On HumanEval-MT, Code Llama-LB improves over Code Llama across underrepresented languages; the 15B and 20B models surpass Code Llama-34B in average scores.The 9B model slightly underperforms Code Llama-13B by 0.2%; Table 2 reports Pass@1 on anonymized HumanEval and HumanEval-MT.Source: §4.3.2 · p. 6, Table 2 · p. 7

4.4.2 Results

• Orca 2-LB-9B improves on Bengali BBH-BN, surpassing Orca 2-13B; gains are notable on subtasks requiring linguistic understanding.The reported examples include CAUSAL JUDGEMENT and SNARKS; Table 3 reports average accuracy across 14 subtasks for English BBH and Bengali BBH-BN.Source: §4.4.2 · p. 7, Table 3 · p. 7

5.1 PCA

• Orca 2-LANGBRIDGE maps all tested languages into one representation cluster, unlike Orca 2’s three clusters.The analysis interprets this as evidence that Henc representations remain relatively language-neutral.Source: §5.1 · p. 8

5.2 Accidental Translations

• Only 8 of 178 SNARK rationales contained accidental third-language translations, while seven included Bengali.The third-language translations appeared in Chinese, Danish, Hindi, Japanese, Marathi, and Turkish.Source: §5.2 · p. 9
• The Bengali CoTs suggest Henc is not perfectly language-agnostic and that the LM sometimes identifies Bengali input.The authors suggest adapting the mT5 encoder for greater language-neutrality as a future direction.Source: §5.2 · p. 9

6.1 HRL Performance Degradation

• LangBridge sometimes underperforms its target LM in high-resource languages, and the causes remain for future investigation.The authors present possible explanations rather than established causes.Source: §6.1 · p. 9
• When the target LM is already proficient in a language, soft-prompt representations may be harder to interpret than its native embeddings.The authors frame this as a conjecture about performance declines.Source: §6.1 · p. 9
• Training used maximum input and output lengths of 1,024 and 128 tokens, considerably shorter than original LM training lengths, especially for Llemma.The authors attribute these caps to limited computing budget.Source: §6.1 · p. 9
• Shared training hyperparameters may be suboptimal for some models, while substitute corpora for Code Llama and Orca 2 introduce distribution shift.Their original training corpora were inaccessible.Source: §6.1 · p. 9

6.2 Multilingual CoT Capabilities?

• English-only training leads the authors to expect predominantly English CoT, which they consider a potentially useful baseline for multilingual reasoning (Shi et al., 2023).Shi et al. report that English CoT consistently leads to competitive results.Source: §6.2 · p. 9
• Adding non-English training data might induce multilingual CoT, but would conflict with the goal of avoiding multilingual reasoning data collection..Source: §6.2 · p. 9

• LangBridge extends language models to multilingual reasoning tasks without multilingual supervision.Source: §7 · p. 9
• The authors report that LangBridge is effective at enhancing reasoning capabilities for low-resource languages.Source: §7 · p. 9
• The paper’s analysis indicates that LangBridge’s effectiveness is due to the language-agnostic nature of multilingual representations.Source: §7 · p. 9

• English-only training means LangBridge models may not be proficient at generating text in languages other than English.Source: Limitations · p. 10
• A noticeable performance gap between high- and low-resource languages remains despite LangBridge narrowing it.Source: Limitations · p. 10
• Reasoning gains for a language depend on both the target LM’s and encoder’s proficiency in that language.Source: Limitations · p. 10

Ethical Considerations

• The datasets are primarily academic, likely contain relatively few offensive or personal examples, but may still include such content; model-use licenses follow the original LMs' terms.Source: Ethical Considerations · p. 10

A.1 XCOPA

• LANGBRIDGE reduces Orca 2's performance degradation across XCOPA languages except Quechua, which is not covered by mT5's 101 languages.The original models approach random-chance accuracy on some underrepresented languages despite near-perfect COPA accuracy.Source: §A.1 · p. 14

A.2 MSVAMP

• MetaMath-LANGBRIDGE models generalize successfully to MSVAMP, an out-of-domain test set not used to train MetaMath or MathOctopus.MSVAMP is a zero-shot multilingual grade-school math dataset translated into 10 languages; no chain-of-thought rationale examples are provided.Source: §A.2 · p. 14

A.3 BBH

• Orca 2-LANGBRIDGE models show noticeable degradation on BBH DATE UNDERSTANDING; the authors observe arbitrary-current-date assumptions in their chain-of-thought and speculate that dataset distribution shift partially contributes.They note that the original Orca 2 training dataset is unavailable, making a thorough ablation of this explanation challenging.Source: §A.3 · p. 14

A.4 HumanEval

• On anonymized HumanEval-MT prompts, Code Llama models decline more sharply than LANGBRIDGE models, suggesting weaker comprehension of natural-language text in another language.The table contrasts original prompts at the top with anonymized prompts at the bottom.Source: §A.4 · p. 14, Table 4 · p. 14

B Inference Throughput of LANGBRIDGE Model

• Orca 2-LANGBRIDGE-9B uses considerably less compute and memory than Orca 2-13B while outperforming it on BBH-BN and XCOPA.Inference throughput is measured over 500 XCOPA examples.Source: B Inference Throughput of LANGBRIDGE… · p. 15, Table 5 · p. 15

C General-domain Language Models

• On MGSM, LANGBRIDGE also enhances low-resource-language performance for general-domain Llama 2 and Mistral 7B models using SlimPajama samples for training.Source: C General-domain Language Models · p. 15

D.1 Freezing/Unfreezing

• Freezing the encoder appears beneficial for finetuned MetaMath and Orca but harms pretrained models; training the language model during alignment does not improve performance.The authors conjecture that dataset entropy may explain the encoder-freezing difference and that in-domain data may explain the lack of gains from LM training.Source: §D.1 · p. 15, Table 6 · p. 15

D.2 Encoder Size

• On XCOPA, performance rises as the mT5 encoder scales from 270M to 2.2B parameters and then saturates through 6.7B; the two smaller encoders underperform the original Orca 2-7B.Scaling beyond 2.2B provides diminishing returns.Source: §D.2 · pp. 15–16, Figure 5 · p. 16

D.3 Training Set Size

• XCOPA performance peaks at 120,000 training examples, 60% of the 200,000-example main-experiment setting.The figure compares performance across five training-set sizes.Source: §D.3 · p. 16, Figure 6 · p. 16

D.4 Different Architectures

• With Orca 2-7B and mT5-XL, an MLP connector slightly decreases performance versus the linear layer, while a perceiver-style resampler yields random performance.The linear layer is the architecture used in the main experiments.Source: §D.4 · p. 16, Table 7 · p. 16

D.5 Different Encoder Models

• Using umT5 or XGLM as the encoder yields random performance, and freezing either makes no difference; the authors speculate that alignment fails with a non-LM Seq2Seq encoder.Further investigation is left for future work.Source: §D.5 · p. 16, Table 8 · p. 16

E Quality Estimation of HumanEval-MT

• Code Llama backtranslation scores generally match original HumanEval, except for Code Llama-34B on Swahili; comparable 7B results and lower automatic scores suggest GPT-4 may struggle with complex Swahili examples.The evaluation uses anonymized function names and docstring-based BLEU and chrF metrics.Source: E Quality Estimation of HumanEval-MT · pp. 16–17, Table 9 · p. 17

F Experimental Details

• LANGBRIDGE training uses AdamW, learning rates of 6e−4 for the linear layer and 2e−5 for the encoder, and an effective batch size of 128; BLOOM baselines instead use 2e−5 throughout and sequence length 1,152.Source: F Experimental Details · p. 17
• Evaluation uses adapted LM Evaluation Harness and Bigcode Evaluation Harness code, reports single runs with a fixed seed, and releases the evaluation code.Source: F Experimental Details · p. 17
• HumanEval-MT is constructed by prompting GPT-4 with two Korean translations by a native Korean author as format-preserving examples for translating target languages.The examples guide GPT-4 to preserve format and translate natural language, not necessarily to improve translation quality.Source: F Experimental Details · p. 17

G CoT Examples

• The chain-of-thought examples include a BBH-BN causal-judgement success, where Orca 2-LANGBRIDGE-13B gained 13.4%, and a date-understanding failure, where it lost 6.4%.Figures show generated outputs for MGSM Telugu and the two Bengali subtasks, with input translations in parentheses.Source: Figure 7 · pp. 17, 19, Figure 8 · p. 20, Figure 9 · p. 21