PaperlayerView exact revision on arXiv

Source-linked AI summary

Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality

Mengyu Bu, Yang Feng

arXiv:2603.17512v4cs.CL

TL;DR

LLMs contain substantial cross-lingual knowledge but struggle with low-resource and unseen languages because that knowledge is difficult to interface with diverse linguistic representations. XBridge composes a multilingual encoder-decoder with a frozen LLM using mapping and optimal-transport alignment, and improves multilingual performance across tasks without retraining the LLM.

  • Problem

    LLMs have strong general reasoning and cross-lingual knowledge but perform imbalancedly across languages, especially struggling with low-resource and unseen languages.

  • Method

    XBridge composes a multilingual encoder, frozen LLM, and multilingual decoder with lightweight mapping layers and optimal transport alignment across heterogeneous representation spaces.

  • Results

    Across four LLMs and multilingual understanding, reasoning, summarization, and generation tasks, XBridge outperforms strong baselines, especially on low-resource and unseen languages, without retraining the LLM.

  • Takeaways & Limitations

    XBridge enables multilingual extension by offloading multilingual capability to an external NMT model while preserving the LLM as a knowledge and reasoning core.

  • Takeaways & Limitations

    XBridge still exhibits some multilingual capability imbalance because the external encoder-decoder and base LLM jointly limit complete uniformity across languages.

Abstract

from arXiv · show

Large language models (LLMs) exhibit strong general intelligence, yet their multilingual performance remains highly imbalanced. Although LLMs encode substantial cross-lingual knowledge in a unified semantic space, they often struggle to reliably interface this knowledge with low-resource or unseen languages. Fortunately, pretrained encoder-decoder translation models already possess balanced multilingual capability, suggesting a natural complement to LLMs. In this work, we propose XBridge, a compositional encoder-LLM-decoder architecture that offloads multilingual understanding and generation to external pretrained translation models, while preserving the LLM as an English-centric core for general knowledge processing. To address the resulting representation misalignment across models, we introduce lightweight cross-model mapping layers and an optimal transport-based alignment objective, enabling fine-grained semantic consistency for multilingual generation. Experiments on four LLMs across multilingual understanding, reasoning, summarization, and generation indicate that XBridge outperforms strong baselines, especially on low-resource and previously unseen languages, without retraining the LLM.

1 Introduction

LLMs possess broad reasoning and cross-lingual knowledge but struggle to interface that knowledge with low-resource and unseen languages. XBridge composes multilingual NMT models with an LLM to align representations and extend multilingual understanding and generation.

  • LLMs perform reliably in English and some high-resource languages but often fail to understand or generate low-resource and unseen languages robustly.
  • The central limitation is interfacing LLM knowledge with diverse linguistic representation spaces, rather than lacking cross-lingual knowledge.
  • Multilingual encoder-decoder NMT models complement LLMs by supporting semantic transfer across hundreds of languages, including many low-resource ones.
  • Existing encoder-augmented approaches improve multilingual understanding but leave generation largely English-centric, while inserting a frozen LLM creates decoder-side semantic misalignment.
  • XBridge uses an encoder-LLM-decoder architecture that aligns heterogeneous representation spaces while preserving the LLM as an English-centric knowledge core.
  • Across four LLMs and multiple multilingual tasks, XBridge improves low-resource and unseen-language performance without retraining the LLM.

2 Related Work

Prior work expands multilingual LLM capability through multilingual data, translation, mixture-of-experts methods, or encoder augmentation. XBridge extends encoder-based integration by composing a multilingual encoder-decoder with an LLM.

  • Data-level multilingual augmentation: Data-level approaches construct multilingual corpora, translate instructions or inputs, and use mixture-of-experts methods for language expansion.
  • Encoder-augmented multilingual LLMs: Encoder-augmented approaches inject pretrained multilingual encoder representations into LLMs to improve cross-lingual understanding and exploit complementary language and reasoning capabilities.
  • XBridge: XBridge composes a pretrained multilingual encoder-decoder with an LLM, broadening the encoder-augmentation approach to multilingual understanding and generation.

3 Method

XBridge composes a multilingual encoder, a frozen English-centric LLM, and a multilingual decoder through lightweight mappings, with optimal transport aligning representations for multilingual generation. A three-stage strategy progressively aligns components, adapts multilingual inputs, and improves decoder-side generation while keeping the LLM frozen.

  • 3.1 Architecture: XBridge uses an encoder-LLM-decoder architecture that connects a pretrained multilingual NMT model and LLM through lightweight cross-model mapping layers.The encoder supplies multilingual representations, the LLM performs knowledge processing, and the decoder generates target-language outputs.
  • 3.1 Architecture: The encoder-side mapping projects multilingual representations into the LLM space, while the decoder-side mapping projects LLM hidden states into the decoder space for cross-attention.The LLM uses penultimate-layer hidden states, which are mapped into decoder key-value representations.
  • 3.2 Optimal Transport-Based Alignment: XBridge uses optimal transport to align encoder representations of LLM outputs with decoder-side LLM representations despite heterogeneous tokenizers and sequence-length mismatch.The objective provides soft, many-to-many token matching and token-level semantic supervision for the decoder-side mapping.
  • 3.2 Optimal Transport-Based Alignment: The OT loss regularizes decoder-side mapping with encoder-derived representations, encouraging semantic compatibility with the multilingual encoder-decoder space and improving multilingual generation quality.The same alignment also facilitates more effective use of multilingual encoder signals by the LLM.
  • 3.3 Three-Stage Training Strategy: XBridge trains in three stages: cross-model mapping with trilingual data, encoder-side task adaptation, and decoder-side adaptation, while keeping the LLM frozen.Stage 1 trains mappings and decoder cross-attention; stage 2 updates only the encoder mapping; stage 3 updates only the decoder mapping and cross-attention layers.
  • 3.3 Three-Stage Training Strategy: The training objectives combine LLM generation, multilingual decoder generation, and optimal transport losses, with different terms activated across stages.Separating encoder- and decoder-side adaptation avoids conflicts between LLM and decoder objectives.

4 Experiment

XBridge is evaluated across four LLMs, ten languages, and multilingual translation, reasoning, summarization, and generation settings. It improves low-resource multilingual performance while preserving high-resource and English capabilities.

  • Experiment Setup: Experiments use four base LLMs, NLLB-200-1.3B, and ten languages spanning low- and high-resource settings.Bengali, Swahili, and Thai are treated as low-resource languages; the remaining seven are high-resource.
  • Experiment Setup: XBridge is compared with multilingual instruction fine-tuning, Translate-Test, MindMerger, and LayAlign baselines.The baselines respectively fine-tune the LLM, translate through English, augment LLM inputs with multilingual encoder representations, or add layer-wise fusion.
  • Experimental Results: XBridge substantially improves multilingual understanding and generation across base LLMs, with especially large gains on low-resource languages.The cross-model mapping is evaluated on FLORES-101 using multilingual understanding and generation measures.
  • Experimental Results: XBridge approaches NLLB-200-1.3B and outperforms encoder-augmented baselines, while high-resource performance remains comparable to the base LLMs.These results indicate that multilingual capability can be offloaded without degrading the original high-resource strengths of the LLM.
  • Experimental Results: Encoder adaptation improves multilingual reasoning over the base LLM, encoder-only baselines, and Translate-Test without sacrificing English-centric reasoning.The reported MGSM gains are attributed to better semantic transfer between multilingual encoder representations and the LLM reasoning space.
  • Experimental Results: Decoder-generated multilingual reasoning reaches English LLM accuracy on MGSM, while XBridge outperforms encoder-augmented baselines and beats SFT on average Rouge-L for XL-Sum.The strongest XL-Sum gains occur in languages where multilingual generation is more challenging.

5 Analysis

XBridge’s ablations show that encoder-decoder collaboration, OT alignment, and stage-wise optimization are central to multilingual reasoning and generation. Additional analyses indicate cross-lingual transfer, language-on-demand generation, backbone generality, and strong dependence of multilingual capability on NMT capacity.

  • Encoder-Decoder Collaboration: Removing the decoder preserves competitive multilingual-to-English understanding but fails multilingual generation and underperforms XBridge on MGSM.The ablation indicates encoder-only augmentation is insufficient for multilingual reasoning and generation.
  • OT Alignment Objectives: Removing OT degrades every benchmark, particularly multilingual generation, highlighting token-level alignment between the LLM and multilingual decoder.OT addresses heterogeneous representation spaces and tokenization differences across model components.
  • Stage-Wise Optimization: Skipping stage 1 causes a substantial drop across metrics, while jointly training stages 2 and 3 underperforms stage-wise optimization.The results support first establishing coarse-grained cross-model alignment before fine-grained encoder and decoder specialization.
  • Generalization to Untuned Languages: XBridge generalizes to 41 untuned languages, approaching external NLLB performance, while OT is especially important for En→X generation.The findings support language-agnostic semantic transfer and robust alignment across heterogeneous tokenizations.
  • Language-on-Demand Generation: Changing the target language enables language-on-demand generation, with performance driven mainly by target language on FLORES-101 and remaining largely stable across output languages on MGSM.The target-language token can be switched without retraining.
  • Representation and Backbone Analysis: OT makes LLM and decoder representations overlap substantially in t-SNE, while XBridge remains effective with M2M100-1.2B and benefits from larger NLLB models mainly on multilingual capability.On MGSM, increasing NLLB size produces only marginal reasoning changes, suggesting stronger dependence on the LLM core.

6 Conclusion

XBridge offloads multilingual capability to an external encoder-decoder NMT model while retaining the LLM as an English-centric knowledge-processing core. The framework raises low-resource and unseen-language performance near external NMT levels without compromising core LLM abilities.

  • 6 Conclusion: XBridge efficiently extends multilingual capability through an external encoder-decoder NMT model while preserving the LLM’s core abilities.The conclusion reports near-external-NMT performance for low-resource and unseen languages without retraining the LLM.

Limitations

XBridge’s relaxed OT alignment addresses heterogeneous tokenizations without positional correspondence, but the combined encoder-decoder and LLM still leave residual multilingual imbalance.

  • The combined external NMT model and base LLM constrain complete uniformity across languages, leaving some multilingual imbalance.Future work could further harmonize the two components.
  • XBridge applies OT because heterogeneous tokenizers produce unequal-length sequences without explicit token-wise correspondence.The alignment therefore operates between representation sequences rather than matching positions directly.
  • Exact OT alignment costs O(n^3) linear programming, while approximate Sinkhorn and IPOT methods still add substantial training overhead.The computational burden motivates a relaxed formulation.
  • Removing the second marginal constraint yields a lower-bound OT objective with a closed-form solution that transports each representation to its most similar counterpart.This supports unequal-length alignments under cosine distance.
  • The relaxed objective anchors decoder-side representations to the frozen multilingual encoder’s semantic geometry, targeting semantic compatibility rather than exact distribution matching.The formulation provides alignment without assuming positional correspondence.

B Details for Training Data

Training data is organized across translation, reasoning, and summarization stages, combining sampled translation pairs with multilingual task datasets and constructed bilingual or trilingual examples.

  • Training Data in Stage 1: Stage 1 samples English-centric OPUS-100 translation pairs with 50k samples per translation direction.XBridge additionally uses NLLB-200-3.3B to translate English into other languages and construct trilingual data.
  • Training Data in Stage 1: Stage 1 trains on y-en-x data, with the encoder processing translated sentences and the decoder processing natural sentences to reduce generation noise.
  • Stage 2 and Stage 3: Multilingual reasoning data contains 30K samples per language across ten languages.
  • Stage 2 and Stage 3: XL-Sum supplies multilingual abstractive summarization data with an upper limit of 30K samples to address its imbalanced language distribution.
  • Data Statistics: Figure 10 provides detailed statistics for the training data used across stages.

C Detailed Results

The detailed-results appendix reports task-specific metrics, ablations, untuned-language generalization, language-on-demand generation, and comparisons across composed model sizes.

  • Main Experiments: Tables 4–7 provide detailed BLEU, COMET, MGSM accuracy, and multilingual Rouge-L results for the main experiments.
  • Generation: Tables 12 and 13 evaluate cross-lingual generation and language-on-demand generation using BLEU and accuracy, respectively.
  • Model Variants: Tables 14–17 compare composed models using M2M100-1.2B and NLLB-200 at 600M versus 1.3B sizes.

D.1 Efficiency Analysis

XBridge adds moderate computational cost for multilingual generation while enabling controllable target-language outputs and stronger multilingual reasoning comparisons across languages.

  • Efficiency: XBridge has limited training overhead, slower inference than LLM-only SFT, and faster inference than the cascaded Translate-Test pipeline.The additional encoder and decoder account for the inference slowdown.
  • Language Control: MindMerger and LayAlign generate English responses, whereas XBridge controls the output language through an explicit decoder language token.Figure 11 compares these multilingual-generation behaviors.
  • Multilingual Commonsense Reasoning: Figure 12 compares XBridge with MindMerger and LayAlign on cross-lingual understanding and logical reasoning across eight languages.
  • Multilingual Commonsense Reasoning: XBridge achieves the highest average accuracy on MetaMath-7B across most languages, with larger gains in non-English languages while preserving English performance.The reported pattern indicates more balanced multilingual reasoning performance.
Loading 2603.17512v4…