Source-linked AI summary
Pre-training via Paraphrasing
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, Luke Zettlemoyer
TL;DR
MARGE addresses the need for a viable alternative to masked language modeling by learning to reconstruct targets from retrieved related documents in multiple languages. It jointly learns retrieval and generation, capturing several document-level tasks and achieving strong zero-shot and fine-tuned performance across languages.
Problem
Masked language modeling is the dominant pre-training paradigm, motivating a viable alternative self-supervision strategy for broad multilingual tasks.
Method
MARGE retrieves related multilingual documents, conditions a sequence-to-sequence model on them, and jointly learns retrieval and reconstruction from the reconstruction loss.
Results
MARGE performs strongly across discriminative and generative tasks in many languages, including zero-shot document translation up to BLEU 35.8.
Takeaways & Limitations
MARGE establishes retrieved-document reconstruction as a viable alternative to masked language modeling and supports tasks with little or no fine-tuning.
Takeaways & Limitations
Performance is inconsistent across languages, and broadening beyond news and Wikipedia may require approximate nearest neighbor search.
Abstract
from arXiv · showhide
We introduce MARGE, a pre-trained sequence-to-sequence model learned with an unsupervised multi-lingual multi-document paraphrasing objective. MARGE provides an alternative to the dominant masked language modeling paradigm, where we self-supervise the reconstruction of target text by retrieving a set of related texts (in many languages) and conditioning on them to maximize the likelihood of generating the original. We show it is possible to jointly learn to do retrieval and reconstruction, given only a random initialization. The objective noisily captures aspects of paraphrase, translation, multi-document summarization, and information retrieval, allowing for strong zero-shot performance on several tasks. For example, with no additional task-specific training we achieve BLEU scores of up to 35.8 for document translation. We further show that fine-tuning gives strong performance on a range of discriminative and generative tasks in many languages, making MARGE the most generally applicable pre-training method to date.
1 Introduction
MARGE presents a pre-training alternative to masked language modeling by reconstructing target text from retrieved, related documents across languages. Its retrieval-conditioned objective jointly learns retrieval and generation while capturing aspects of paraphrasing, translation, summarization, and information retrieval.
- MARGE replaces masked-token reconstruction with self-supervised paraphrasing of related document collections in many languages.
- The model retrieves related texts, separately encodes them, and generates the target while using relevance scores to bias cross-attention.The reconstruction loss jointly trains the sequence-to-sequence and retrieval components.
- Retrieved documents may differ lexically and linguistically from the target, making retrieval a diverse denoising bottleneck that emphasizes paraphrasing over memorization.
- Without fine-tuning, MARGE captures elements of paraphrasing, translation, multi-document summarization, and information retrieval, including document translation up to BLEU 35.8.With fine-tuning, it achieves competitive performance with masked language models across discriminative and generative tasks in many languages.
2 Model
MARGE jointly learns document relevance and target reconstruction by retrieving related evidence, scoring it, and conditioning a multilingual sequence-to-sequence decoder on the retrieved documents. Its reconstruction loss updates both the generator and relevance model, while retrieval batches are constructed from related-document graphs.
- Relevance scores: MARGE scores every target–evidence document pair with cosine similarity from a shared document encoder.The encoder maps token lists to fixed-size representations and is applied to both target and evidence documents.
- Joint learning: Backpropagating reconstruction loss jointly improves the sequence-to-sequence model and the relevance model because more relevant evidence raises reconstruction likelihood.The relevance function is also reused when constructing batches of related documents.
- Training dynamics: Training succeeds from random initialization despite a retrieval–reconstruction chicken-and-egg problem.Shared encoders make lexical-overlap similarities meaningful enough at initialization to support learning.
- Reconstruction model: The model reconstructs each target from retrieved evidence documents and their relevance scores through an autoencoder bottleneck.This bottleneck indirectly conditions reconstruction on the target through retrieved documents and scores rather than direct copying.
- Reconstruction model: The reconstruction model encodes retrieved documents separately, concatenates their representations, and uses relevance scores to bias decoder cross-attention.The trainable scalar β controls how strongly document similarity influences attention.
- Batch construction: Batches are built from related-document graphs, emphasizing cross-language edges and maximizing connectivity under the available token budget.Documents are organized into related shards, similarities are thresholded, and a local search selects high-weight subgraphs.
3 Training
Training uses a large Transformer-based sequence-to-sequence architecture, multilingual document preprocessing, and staged pre-training on news followed by Wikipedia. The setup also specifies retrieval indexing, language control, and task-specific fine-tuning procedures.
- Architecture: The model uses a 12-layer Transformer encoder with dimension 1024 and a decoder whose feed-forward layers are expanded to 16536.The decoder also adds four self-attention-only Transformer layers at its base.
- Pre-training: Pre-training proceeds in two stages: 450k steps with 64 workers, followed by 550k steps with 2048 workers.The learning rate is annealed separately in each stage, with 10k warmup steps in the first stage.
- Pre-training: MARGE is further pre-trained on Wikipedia for 100k steps, with retrieval indexing rebuilt every 10k updates.Retrieval thresholds yield an average of four monolingual and four crosslingual links per target document.
- Data Pre-processing: Documents are deduplicated, language-identified across 26 languages, and divided into chunks of length 512.All chunks may serve as evidence documents, while news targets are restricted to the first chunk of each document.
- Fine-tuning: Fine-tuning feeds generation inputs to the encoder and uses encoder-decoder representations for classification tasks.For zero-shot transfer, word embeddings and the first four decoder layers are frozen.
4 Experiments
MARGE is evaluated across multilingual retrieval, translation, summarization, paraphrase detection, and question answering, including zero-shot and fine-tuned settings. It performs strongly across these tasks, while results vary by language and generation direction.
- Cross-lingual sentence retrieval: MARGE outperforms other unsupervised models by almost 10 points on BUCC cross-lingual sentence retrieval.
- Document-level machine translation: 35.8 BLEU is achieved for German-to-English document translation without bitext, while performance varies considerably by language.Arabic-to-English is also strong, whereas Japanese performs less well and non-English generation is harder, particularly for non-Latin scripts.
- Document-level machine translation: MARGE and mBART perform similarly on supervised document translation, with MARGE better on English-German and mBART better on Chinese-English.Both outperform baselines by a wide margin.
- Summarization: MARGE outperforms extractive mBERT and a non-pre-trained sequence-to-sequence model on MLSum summarization.Zero-shot summarization outperforms a strong lead-3 baseline and a supervised pointer-generator model on Spanish and Russian.
- Paraphrase detection: MARGE edges out a new state of the art on PAWS-X paraphrase detection under zero-shot transfer to other languages.
- Question answering: MARGE achieves competitive MLQA performance with XLM-R, sets the state of the art for Chinese, and outperforms other models by a wide margin.
5 Analysis
MARGE’s analysis examines whether its outputs reflect retrieved multilingual information rather than memorized text, and how retrieval patterns vary across languages. The results show meaningful cross-lingual retrieval but also uneven language coverage and domain-dependent limitations.
- Reconstruction behavior: Model outputs vary substantially for the same topic depending on the input, indicating that MARGE is not simply memorizing text.Most output facts are supported by the input, although some important source facts are omitted.
- Reconstruction behavior: MARGE often produces outputs closely aligned with inputs despite not being trained on literal translations.A possible explanation is that literal translations represent one mode within a diverse paraphrase distribution.
- Retrieval behavior: Retrieval statistics vary with corpus frequency, linguistic relatedness, topic overlap, and feedback loops between reconstruction and retrieval.When reconstruction cannot translate between languages, the retrieval model may learn that documents in those languages are less relevant.
- Retrieval behavior: All languages retrieve the highest proportion of documents within their own language, while remaining retrievals are distributed across several other languages.The retrieval matrix shows stronger affinities between geographically or linguistically related languages, including Bulgarian–Russian and Chinese–Japanese.
- Retrieval behavior: Retrieval is weakest for some Indo-Iranian languages and languages represented by only one family member, reducing pre-training to independent language updates.This resembles the multilingual-training behavior of mBART, mBERT, and XLM for those cases.
- Overall findings: MARGE achieves strong performance across discriminative and generative tasks, including translation with BLEU scores as high as 35.8.The discussion reports competitive results with less general models, including XLM-R, despite significantly greater pre-training resources for XLM-R.
- Limitations: The experiments are limited by training on news and Wikipedia, batch-size requirements above one, and inconsistent performance across languages.Broader domains may require approximate nearest-neighbor search, while substantially larger models would require model-parallel training.
6 Related Work
MARGE departs from dominant masked-language-model pre-training by reconstructing complete target documents from retrieved multilingual evidence. It differs from bitext mining and other cross-lingual approaches by using noisy document-level paraphrases without manually translated sentences.
- NLP pre-training: MARGE differs from masked language models by using complete, uncorrupted text during pre-training rather than reconstructing removed input spans.Its self-supervision comes from learning to paraphrase collections of related documents in many languages.
- Bitext mining: Unlike bitext mining, MARGE retrieves noisy document-level paraphrases rather than literal parallel sentences.The approach aims to learn linguistic information from loose cross-lingual correspondence.
- Cross-lingual learning: Where prior cross-lingual methods use multilingual MLMs or machine-translation systems, MARGE pre-trains on loose cross-lingual paraphrases.This positions retrieval-based paraphrasing as a distinct cross-lingual pre-training objective.
- Language models with retrieval: Retrieval-based language-modeling work improves prediction by retrieving relevant evidence, whereas MARGE uses retrieved documents to reconstruct the original target.Its objective combines retrieval with document-level sequence-to-sequence reconstruction.
7 Conclusion
The paper concludes that MARGE is a viable alternative to masked language modeling, using retrieved documents to reconstruct originals. It performs strongly across many tasks and languages, while future work should extend its domains, languages, and task alignment.
- Conclusion: MARGE uses retrieved documents to reconstruct the original as an alternative pre-training approach for language understanding and generation.
- Conclusion: MARGE performs strongly on discriminative and generative tasks in many languages, both with and without fine-tuning.
- Conclusion: Future work should scale MARGE to more domains and languages and align its pre-training objectives more closely with different end tasks.
A Additional Results
The Tatoeba results report zero-shot sentence retrieval performance comparable to XLM-R, alongside substantial variation across languages.
- Tatoeba zero-shot retrieval: MARGE performs comparably to XLM-R on Tatoeba zero-shot sentence retrieval, with significant variation across languages.The table includes only languages present in all models’ pre-training data.
B Pre-training Data
Pre-training data spans languages from multiple families and geographical regions.
- B Pre-training Data: The pre-training corpus includes documents across Germanic, Hellenic, Romance, Slavic, and Indo-Iranian language families.These families represent a range of linguistic families and geographical regions.